Observability-Driven Development

The Shift: Observability as a First-Class Concern

After two years of retrofitting OpenTelemetry into legacy systems, I learned the hard way: observability should be part of the design, not an afterthought.

Here's what changed when I started building systems with observability as a core requirement from day one.

Before vs. After

Before (Observability as Afterthought)

// Write code first
export async function processOrder(orderId: string) {
  const order = await db.query('SELECT * FROM orders WHERE id = $1', [orderId]);
  const inventory = await checkInventory(order.items);
  const payment = await processPayment(order.total);
  const shipping = await createShipment(order);
  return { success: true };
}

// Add observability later (painful)
export async function processOrder(orderId: string) {
  const span = tracer.startSpan('processOrder');
  try {
    span.setAttribute('order.id', orderId);
    const order = await db.query('SELECT * FROM orders WHERE id = $1', [orderId]);
    span.setAttribute('order.total', order.total);
    
    // Forgot to add span for this!
    const inventory = await checkInventory(order.items);
    
    const paymentSpan = tracer.startSpan('processPayment');
    const payment = await processPayment(order.total);
    paymentSpan.end();
    
    // This needs its own span too...
    const shipping = await createShipment(order);
    
    span.setStatus({ code: SpanStatusCode.OK });
    return { success: true };
  } catch (error) {
    span.setStatus({ code: SpanStatusCode.ERROR });
    span.recordException(error);
    throw error;
  } finally {
    span.end();
  }
}

After (Observability-First)

import { trace, context } from '@opentelemetry/api';
import { SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service');

// Design with observability from the start
export async function processOrder(orderId: string) {
  return tracer.startActiveSpan('processOrder', async (span) => {
    span.setAttribute('order.id', orderId);
    
    try {
      // Each step is automatically instrumented
      const order = await fetchOrder(orderId);
      span.setAttribute('order.total', order.total);
      span.setAttribute('order.items_count', order.items.length);
      
      await validateInventory(order);
      await chargePayment(order);
      await scheduleShipment(order);
      
      span.setStatus({ code: SpanStatusCode.OK });
      return { success: true, orderId };
    } catch (error) {
      span.setStatus({ 
        code: SpanStatusCode.ERROR,
        message: error.message 
      });
      span.recordException(error);
      throw error;
    } finally {
      span.end();
    }
  });
}

// Each helper function also includes spans
async function validateInventory(order: Order) {
  return tracer.startActiveSpan('validateInventory', async (span) => {
    span.setAttribute('items_count', order.items.length);
    
    for (const item of order.items) {
      const available = await checkStock(item.sku);
      span.addEvent('stock_checked', {
        sku: item.sku,
        available,
        needed: item.quantity,
      });
      
      if (!available) {
        span.setStatus({ 
          code: SpanStatusCode.ERROR,
          message: `Insufficient stock for ${item.sku}` 
        });
        throw new Error(`Out of stock: ${item.sku}`);
      }
    }
    
    span.end();
  });
}

Key Differences:

1. Spans are part of the function design - Not bolted on later

2. Consistent attribute naming - Follows semantic conventions from day one

3. Meaningful events - Business context built in

4. Error context is rich - Failures tell the full story

Design Patterns for Observability

1. Observability Decorators

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service');

// Decorator for automatic instrumentation
function Traced(operationName?: string) {
  return function (
    target: any,
    propertyKey: string,
    descriptor: PropertyDescriptor
  ) {
    const originalMethod = descriptor.value;
    
    descriptor.value = async function (...args: any[]) {
      const spanName = operationName || `${target.constructor.name}.${propertyKey}`;
      
      return tracer.startActiveSpan(spanName, async (span) => {
        try {
          // Add method arguments as attributes
          span.setAttribute('args', JSON.stringify(args));
          
          const result = await originalMethod.apply(this, args);
          
          span.setStatus({ code: SpanStatusCode.OK });
          return result;
        } catch (error) {
          span.setStatus({ 
            code: SpanStatusCode.ERROR,
            message: error.message 
          });
          span.recordException(error);
          throw error;
        } finally {
          span.end();
        }
      });
    };
    
    return descriptor;
  };
}

// Usage
class OrderService {
  @Traced()
  async createOrder(userId: string, items: CartItem[]) {
    // Business logic
    const order = await this.db.insert('orders', { userId, items });
    return order;
  }
  
  @Traced('custom-payment-processing')
  async processPayment(orderId: string, amount: number) {
    // Business logic
    const payment = await this.paymentGateway.charge(amount);
    return payment;
  }
}

2. Observable Domain Models

import { trace, context } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service');

class Order {
  constructor(
    public id: string,
    public userId: string,
    public items: OrderItem[],
    public status: OrderStatus
  ) {}
  
  // Each state transition is instrumented
  async submit() {
    return tracer.startActiveSpan('Order.submit', async (span) => {
      span.setAttribute('order.id', this.id);
      span.setAttribute('order.total', this.calculateTotal());
      
      // Validate
      await this.validate();
      
      // Reserve inventory
      await this.reserveInventory();
      
      // Update status
      this.status = 'SUBMITTED';
      await this.save();
      
      span.addEvent('order_submitted', {
        order_id: this.id,
        total: this.calculateTotal(),
      });
      
      span.end();
      return this;
    });
  }
  
  async cancel(reason: string) {
    return tracer.startActiveSpan('Order.cancel', async (span) => {
      span.setAttribute('order.id', this.id);
      span.setAttribute('cancellation.reason', reason);
      
      // Release inventory
      await this.releaseInventory();
      
      // Refund payment
      if (this.isPaid()) {
        await this.refundPayment();
      }
      
      this.status = 'CANCELLED';
      await this.save();
      
      span.addEvent('order_cancelled', {
        order_id: this.id,
        reason,
      });
      
      span.end();
      return this;
    });
  }
  
  private async validate() {
    return tracer.startActiveSpan('Order.validate', async (span) => {
      // Validation logic
      if (this.items.length === 0) {
        span.setStatus({ 
          code: SpanStatusCode.ERROR,
          message: 'Order has no items' 
        });
        throw new Error('Order has no items');
      }
      
      span.end();
    });
  }
}

3. Context Propagation Helpers

import { context, propagation } from '@opentelemetry/api';
import { W3CTraceContextPropagator } from '@opentelemetry/core';

// Helper to extract context from HTTP headers
export function extractContext(headers: Record<string, string>) {
  const propagator = new W3CTraceContextPropagator();
  return propagation.extract(context.active(), headers);
}

// Helper to inject context into outgoing requests
export function injectContext(headers: Record<string, string>) {
  const propagator = new W3CTraceContextPropagator();
  propagation.inject(context.active(), headers);
  return headers;
}

// Usage in Express middleware
app.use((req, res, next) => {
  const ctx = extractContext(req.headers);
  context.with(ctx, () => {
    tracer.startActiveSpan('http.request', (span) => {
      span.setAttribute('http.method', req.method);
      span.setAttribute('http.url', req.url);
      
      res.on('finish', () => {
        span.setAttribute('http.status_code', res.statusCode);
        span.end();
      });
      
      next();
    });
  });
});

Testing with Observability

In-Memory Span Exporter for Tests

import { InMemorySpanExporter } from '@opentelemetry/sdk-trace-base';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';

describe('OrderService', () => {
  let spanExporter: InMemorySpanExporter;
  let provider: NodeTracerProvider;
  
  beforeEach(() => {
    spanExporter = new InMemorySpanExporter();
    provider = new NodeTracerProvider();
    provider.addSpanProcessor(new SimpleSpanProcessor(spanExporter));
    provider.register();
  });
  
  afterEach(() => {
    spanExporter.reset();
  });
  
  it('should create spans for order submission', async () => {
    const service = new OrderService();
    await service.createOrder('user-123', [
      { sku: 'WIDGET-1', quantity: 2 },
    ]);
    
    const spans = spanExporter.getFinishedSpans();
    
    expect(spans).toHaveLength(4);
    expect(spans[0].name).toBe('OrderService.createOrder');
    expect(spans[1].name).toBe('Order.validate');
    expect(spans[2].name).toBe('Order.reserveInventory');
    expect(spans[3].name).toBe('Order.save');
    
    expect(spans[0].attributes['order.total']).toBe(200);
  });
  
  it('should record exceptions in spans', async () => {
    const service = new OrderService();
    
    await expect(
      service.createOrder('user-123', [])
    ).rejects.toThrow('Order has no items');
    
    const spans = spanExporter.getFinishedSpans();
    const validationSpan = spans.find(s => s.name === 'Order.validate');
    
    expect(validationSpan?.status.code).toBe(SpanStatusCode.ERROR);
    expect(validationSpan?.events).toContainEqual(
      expect.objectContaining({
        name: 'exception',
        attributes: expect.objectContaining({
          'exception.message': 'Order has no items',
        }),
      })
    );
  });
});

Trace Assertions

// Custom matchers for span testing
expect.extend({
  toHaveSpan(received: InMemorySpanExporter, spanName: string) {
    const spans = received.getFinishedSpans();
    const found = spans.find(s => s.name === spanName);
    
    return {
      pass: !!found,
      message: () => 
        found 
          ? `Expected not to find span "${spanName}"` 
          : `Expected to find span "${spanName}"`,
    };
  },
  
  toHaveSpanWithAttributes(
    received: InMemorySpanExporter, 
    spanName: string, 
    attributes: Record<string, any>
  ) {
    const spans = received.getFinishedSpans();
    const found = spans.find(
      s => s.name === spanName && 
      Object.entries(attributes).every(([key, value]) => s.attributes[key] === value)
    );
    
    return {
      pass: !!found,
      message: () => 
        found 
          ? `Expected not to find span "${spanName}" with attributes` 
          : `Expected to find span "${spanName}" with attributes ${JSON.stringify(attributes)}`,
    };
  },
});

// Usage
it('should track payment processing', async () => {
  await service.processPayment('order-123', 100);
  
  expect(spanExporter).toHaveSpan('processPayment');
  expect(spanExporter).toHaveSpanWithAttributes('processPayment', {
    'order.id': 'order-123',
    'payment.amount': 100,
  });
});

Documentation Through Observability

Self-Documenting Traces

// Use span events as inline documentation
async function processComplexWorkflow(orderId: string) {
  return tracer.startActiveSpan('processComplexWorkflow', async (span) => {
    span.addEvent('workflow_started', {
      order_id: orderId,
      description: 'Beginning multi-step order fulfillment process',
    });
    
    // Step 1: Fraud check
    span.addEvent('step_1_fraud_check', {
      description: 'Validating order against fraud detection rules',
    });
    const fraudScore = await checkFraud(orderId);
    span.setAttribute('fraud.score', fraudScore);
    
    if (fraudScore > 0.8) {
      span.addEvent('fraud_detected', {
        score: fraudScore,
        action: 'Flagging order for manual review',
      });
      await flagForReview(orderId);
      span.end();
      return { status: 'PENDING_REVIEW' };
    }
    
    // Step 2: Inventory reservation
    span.addEvent('step_2_inventory', {
      description: 'Reserving items from warehouse inventory',
    });
    await reserveInventory(orderId);
    
    // Step 3: Payment processing
    span.addEvent('step_3_payment', {
      description: 'Charging customer payment method',
    });
    await chargePayment(orderId);
    
    span.addEvent('workflow_completed', {
      description: 'Order successfully processed and ready for shipment',
    });
    
    span.end();
    return { status: 'COMPLETED' };
  });
}

Result in Jaeger:

Span: processComplexWorkflow (2.3s)
├─ Event: workflow_started
│  └─ description: "Beginning multi-step order fulfillment process"
├─ Event: step_1_fraud_check
│  └─ description: "Validating order against fraud detection rules"
├─ Attribute: fraud.score = 0.12
├─ Event: step_2_inventory
│  └─ description: "Reserving items from warehouse inventory"
├─ Event: step_3_payment
│  └─ description: "Charging customer payment method"
└─ Event: workflow_completed
   └─ description: "Order successfully processed and ready for shipment"

Performance Budgets

Setting SLOs with Metrics

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('order-service');

// Track against SLO
const requestDuration = meter.createHistogram('http.server.request.duration', {
  description: 'HTTP request duration',
  unit: 'ms',
});

// SLO: 95% of requests < 500ms
const SLO_THRESHOLD = 500;

app.use((req, res, next) => {
  const start = Date.now();
  
  res.on('finish', () => {
    const duration = Date.now() - start;
    
    requestDuration.record(duration, {
      'http.method': req.method,
      'http.route': req.route?.path,
      'http.status_code': res.statusCode,
    });
    
    // Alert if SLO violated
    if (duration > SLO_THRESHOLD) {
      console.warn(`SLO violation: ${req.method} ${req.url} took ${duration}ms`);
    }
  });
  
  next();
});

Prometheus Alerting

groups:
- name: slo_alerts
  interval: 30s
  rules:
  # Alert if p95 exceeds 500ms for 5 minutes
  - alert: HighLatency
    expr: |
      histogram_quantile(0.95,
        sum(rate(http_server_request_duration_bucket[5m])) by (le)
      ) > 500
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High latency detected"
      description: "P95 latency is {{ $value }}ms (SLO: 500ms)"
  
  # Alert if error rate exceeds 1%
  - alert: HighErrorRate
    expr: |
      sum(rate(http_server_request_duration_count{http_status_code=~"5.."}[5m]))
      /
      sum(rate(http_server_request_duration_count[5m]))
      > 0.01
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High error rate detected"
      description: "Error rate is {{ $value | humanizePercentage }}"

Real Production Example

Here's a complete service designed with observability first:

import express from 'express';
import { trace, metrics, SpanStatusCode } from '@opentelemetry/api';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const tracer = trace.getTracer('order-service', '1.0.0');
const meter = metrics.getMeter('order-service', '1.0.0');

// Metrics
const ordersCreated = meter.createCounter('orders.created', {
  description: 'Total orders created',
});

const orderDuration = meter.createHistogram('orders.processing.duration', {
  description: 'Order processing duration',
  unit: 'ms',
});

// Initialize OpenTelemetry
const sdk = new NodeSDK({
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

const app = express();
app.use(express.json());

// POST /orders
app.post('/orders', async (req, res) => {
  return tracer.startActiveSpan('POST /orders', async (span) => {
    const start = Date.now();
    
    try {
      span.setAttribute('user.id', req.body.userId);
      span.setAttribute('items.count', req.body.items.length);
      
      // Validate
      await validateOrder(req.body);
      
      // Create order
      const order = await createOrder(req.body);
      
      // Update metrics
      ordersCreated.add(1, {
        'user.type': req.body.userType,
      });
      
      orderDuration.record(Date.now() - start, {
        'order.status': 'success',
      });
      
      span.setStatus({ code: SpanStatusCode.OK });
      res.json({ success: true, orderId: order.id });
    } catch (error) {
      span.setStatus({ 
        code: SpanStatusCode.ERROR,
        message: error.message 
      });
      span.recordException(error);
      
      orderDuration.record(Date.now() - start, {
        'order.status': 'error',
      });
      
      res.status(500).json({ error: error.message });
    } finally {
      span.end();
    }
  });
});

async function validateOrder(data: any) {
  return tracer.startActiveSpan('validateOrder', async (span) => {
    if (!data.userId) {
      span.setStatus({ 
        code: SpanStatusCode.ERROR,
        message: 'Missing userId' 
      });
      throw new Error('Missing userId');
    }
    
    if (!data.items || data.items.length === 0) {
      span.setStatus({ 
        code: SpanStatusCode.ERROR,
        message: 'No items in order' 
      });
      throw new Error('No items in order');
    }
    
    span.end();
  });
}

async function createOrder(data: any) {
  return tracer.startActiveSpan('createOrder', async (span) => {
    // Simulate database insert
    const order = {
      id: `order-${Date.now()}`,
      userId: data.userId,
      items: data.items,
      createdAt: new Date(),
    };
    
    span.setAttribute('order.id', order.id);
    span.addEvent('order_created', {
      order_id: order.id,
      total_items: order.items.length,
    });
    
    span.end();
    return order;
  });
}

app.listen(3000, () => {
  console.log('Order service running on port 3000');
});

Key Takeaways

1. Design APIs with tracing in mind - Each operation should be a span

2. Use decorators/wrappers - Don't duplicate instrumentation code

3. Test your telemetry - Spans are part of your API contract

4. Document with events - Let traces tell the story

5. Set performance budgets - SLOs backed by metrics

6. Observability is not optional - It's part of the feature

Conclusion

Building systems with observability from day one changed how I develop software:

• Faster debugging: Instrumentation is already there when things break

• Better designs: Thinking in traces leads to cleaner code boundaries

• Living documentation: Traces show how the system actually works

• Confidence in production: Comprehensive visibility from launch day

You've completed the OpenTelemetry 101 series! 🎉

You now have everything you need to:

• ✅ Instrument TypeScript/Node.js services

• ✅ Collect traces, metrics, and logs

• ✅ Deploy to production at scale

• ✅ Integrate with any backend

• ✅ Build observable systems from the ground up

Go forth and make your systems observable!

---

Previous: ← Multi-Backend Integration | Series Home: OpenTelemetry 101 Index

Observability is not a feature, it's a philosophy.