Production Deployment

From 10 to 10,000 Requests Per Second

When I first deployed OpenTelemetry in production, we handled 10 requests/second. The setup was simple:

Single Jaeger instance
One OpenTelemetry Collector
Services sending directly to Collector

Fast forward 6 months: 10,000 requests/second. The original setup collapsed:

Jaeger ran out of storage
Collector became a bottleneck
Export failures caused memory leaks
Traces were getting dropped

This article covers how I scaled OpenTelemetry to handle massive production load.

Architecture Evolution

Phase 1: Simple (0-100 req/s)

┌─────────┐     ┌───────────┐     ┌─────────┐
│ Service │────▶│ Collector │────▶│ Jaeger  │
└─────────┘     └───────────┘     └─────────┘

Works for: Development, small apps, proof-of-concept

Phase 2: Distributed (100-1,000 req/s)

┌─────────┐                         ┌─────────┐
│ Service │──┐                   ┌─▶│ Jaeger  │
└─────────┘  │  ┌───────────┐   │  └─────────┘
             ├─▶│ Collector │───┤
┌─────────┐  │  └───────────┘   │  ┌───────────┐
│ Service │──┘                   └─▶│Prometheus │
└─────────┘                         └───────────┘

Works for: Production apps, small teams

Phase 3: Scaled (1,000-10,000+ req/s)

┌─────────┐                         ┌────────────┐     ┌─────────┐
│ Service │──┐                   ┌─▶│ Collector  │────▶│ Jaeger  │
└─────────┘  │  ┌───────────┐   │  │ (Gateway)  │     └─────────┘
             ├─▶│  sidecar  │───┤  └────────────┘
┌─────────┐  │  │ Collector │   │                     ┌───────────┐
│ Service │──┘  └───────────┘   │  ┌────────────┐    │Prometheus │
└─────────┘                      └─▶│ Collector  │───▶│  (TSDB)   │
                                    │ (Gateway)  │    └───────────┘
                                    └────────────┘

Works for: High-scale production, large teams

Production Kubernetes Deployment

Collector DaemonSet (Sidecar Pattern)

Deploy Collector on every node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: observability
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector:0.91.0
        command:
        - "/otelcol"
        - "--config=/conf/otel-collector-config.yaml"
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 200m
            memory: 400Mi
        ports:
        - containerPort: 4317  # OTLP gRPC
          hostPort: 4317
          protocol: TCP
        - containerPort: 4318  # OTLP HTTP
          hostPort: 4318
          protocol: TCP
        - containerPort: 8888  # Metrics
          protocol: TCP
        volumeMounts:
        - name: config
          mountPath: /conf
        livenessProbe:
          httpGet:
            path: /
            port: 13133
        readinessProbe:
          httpGet:
            path: /
            port: 13133
      volumes:
      - name: config
        configMap:
          name: otel-collector-config
          items:
          - key: otel-collector-config.yaml
            path: otel-collector-config.yaml

Gateway Collector Deployment

Centralized collectors for processing:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector-gateway
  namespace: observability
spec:
  replicas: 3  # High availability
  selector:
    matchLabels:
      app: otel-collector-gateway
  template:
    metadata:
      labels:
        app: otel-collector-gateway
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector:0.91.0
        command:
        - "/otelcol"
        - "--config=/conf/otel-collector-gateway-config.yaml"
        resources:
          limits:
            cpu: 2000m
            memory: 4Gi
          requests:
            cpu: 500m
            memory: 1Gi
        ports:
        - containerPort: 4317
        - containerPort: 4318
        - containerPort: 8888
        volumeMounts:
        - name: config
          mountPath: /conf
      volumes:
      - name: config
        configMap:
          name: otel-collector-gateway-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector-gateway
  namespace: observability
spec:
  selector:
    app: otel-collector-gateway
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318
  type: ClusterIP

DaemonSet Configuration

Node collectors forward to gateway:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    
    processors:
      batch:
        timeout: 10s
        send_batch_size: 1024
      
      memory_limiter:
        check_interval: 1s
        limit_mib: 512
        spike_limit_mib: 128
    
    exporters:
      # Forward to gateway collectors
      otlp:
        endpoint: otel-collector-gateway.observability.svc.cluster.local:4317
        tls:
          insecure: true
        sending_queue:
          enabled: true
          num_consumers: 10
          queue_size: 1000
        retry_on_failure:
          enabled: true
          initial_interval: 1s
          max_interval: 30s
    
    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [otlp]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [otlp]
    
    extensions:
      health_check:
        endpoint: 0.0.0.0:13133

Gateway Configuration

Gateway does heavy processing:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-gateway-config
  namespace: observability
data:
  otel-collector-gateway-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    
    processors:
      batch:
        timeout: 10s
        send_batch_size: 2048
      
      memory_limiter:
        check_interval: 1s
        limit_mib: 2048
        spike_limit_mib: 512
      
      # Tail-based sampling (expensive, do at gateway)
      tail_sampling:
        decision_wait: 10s
        num_traces: 100000
        expected_new_traces_per_sec: 5000
        policies:
          - name: error-policy
            type: status_code
            status_code:
              status_codes: [ERROR]
          - name: slow-policy
            type: latency
            latency:
              threshold_ms: 2000
          - name: probabilistic-policy
            type: probabilistic
            probabilistic:
              sampling_percentage: 1
      
      # Redact sensitive data
      attributes/redact:
        actions:
          - key: password
            action: delete
          - key: api_key
            action: delete
          - key: http.request.header.authorization
            action: delete
    
    exporters:
      # Jaeger
      otlp/jaeger:
        endpoint: jaeger-collector.observability.svc.cluster.local:4317
        tls:
          insecure: true
      
      # Prometheus
      prometheusremotewrite:
        endpoint: http://prometheus.observability.svc.cluster.local:9090/api/v1/write
    
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors:
            - memory_limiter
            - tail_sampling
            - attributes/redact
            - batch
          exporters: [otlp/jaeger]
        
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheusremotewrite]

Jaeger High Availability

Elasticsearch Backend

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-collector
  namespace: observability
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jaeger-collector
  template:
    metadata:
      labels:
        app: jaeger-collector
    spec:
      containers:
      - name: jaeger-collector
        image: jaegertracing/jaeger-collector:1.52
        env:
        - name: SPAN_STORAGE_TYPE
          value: elasticsearch
        - name: ES_SERVER_URLS
          value: http://elasticsearch.observability.svc.cluster.local:9200
        - name: ES_NUM_SHARDS
          value: "5"
        - name: ES_NUM_REPLICAS
          value: "1"
        - name: COLLECTOR_QUEUE_SIZE
          value: "8000"
        - name: COLLECTOR_NUM_WORKERS
          value: "100"
        resources:
          limits:
            cpu: 2000m
            memory: 4Gi
          requests:
            cpu: 500m
            memory: 1Gi
        ports:
        - containerPort: 14250  # gRPC
        - containerPort: 14268  # HTTP
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger-collector
  namespace: observability
spec:
  selector:
    app: jaeger-collector
  ports:
  - name: grpc
    port: 14250
    targetPort: 14250
  - name: http
    port: 14268
    targetPort: 14268

Monitoring the Monitoring System

Critical: Monitor your observability infrastructure!

Collector Metrics

# ServiceMonitor for Prometheus Operator
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: otel-collector
  namespace: observability
spec:
  selector:
    matchLabels:
      app: otel-collector
  endpoints:
  - port: metrics
    interval: 30s

Key metrics to alert on:

groups:
- name: otel-collector
  rules:
  - alert: CollectorHighMemory
    expr: container_memory_usage_bytes{pod=~"otel-collector.*"} / container_spec_memory_limit_bytes > 0.8
    for: 5m
    annotations:
      summary: "Collector memory usage > 80%"
  
  - alert: CollectorExportFailures
    expr: rate(otelcol_exporter_send_failed_spans[5m]) > 100
    for: 2m
    annotations:
      summary: "Collector failing to export spans"
  
  - alert: CollectorQueueFull
    expr: otelcol_exporter_queue_size / otelcol_exporter_queue_capacity > 0.9
    for: 5m
    annotations:
      summary: "Collector export queue > 90% full"

Application Configuration

Services send to local DaemonSet collector:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';

// In Kubernetes, use node-local collector
const collectorEndpoint = process.env.KUBERNETES_SERVICE_HOST
  ? `http://${process.env.NODE_IP}:4317`  // DaemonSet on same node
  : 'http://localhost:4317';  // Local development

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: collectorEndpoint,
  }),
  
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: collectorEndpoint,
    }),
    exportIntervalMillis: 60000,
  }),
});

sdk.start();

console.log(`Sending telemetry to: ${collectorEndpoint}`);

Kubernetes Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  template:
    spec:
      containers:
      - name: order-service
        image: myregistry/order-service:v1.2.3
        env:
        # Inject node IP for node-local collector
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://$(NODE_IP):4317"
        - name: OTEL_SERVICE_NAME
          value: "order-service"

Cost Optimization

Storage Costs

At 10,000 req/s with 1% sampling:

100 traces/second × 15KB avg = 1.5 MB/s
Daily: 130 GB
7-day retention: 910 GB
Monthly: ~3.9 TB

At $0.10/GB Elasticsearch: ~$390/month just for storage

Optimization:

# Jaeger with index lifecycle management
apiVersion: v1
kind: ConfigMap
metadata:
  name: jaeger-config
data:
  config.yaml: |
    storage:
      type: elasticsearch
      options:
        servers: http://elasticsearch:9200
        index-prefix: jaeger
        # Delete indices after 7 days
        max-span-age: 168h
        num-shards: 5
        num-replicas: 1
        # Use rollover for efficient storage
        create-index-templates: true
        use-aliases: true

Runbook: Common Issues

Issue 1: Export Queue Full

Symptoms: Memory usage increasing, spans being dropped

Check:

kubectl logs -n observability otel-collector-xyz | grep "queue is full"

Fix:

# Increase queue size or workers
exporters:
  otlp:
    sending_queue:
      queue_size: 5000  # Increased from 1000
      num_consumers: 20  # Increased from 10

Issue 2: Jaeger Storage Full

Symptoms: Collector export errors, "index read-only"

Check:

curl http://elasticsearch:9200/_cluster/health
# Check disk usage

Fix:

# Delete old indices
curator_cli --host elasticsearch delete_indices --filter_list \
  '[{"filtertype":"age","source":"name","direction":"older","unit":"days","unit_count":7}]'

Issue 3: High Collector CPU

Symptoms: Collector CPU > 80%

Check:

rate(process_cpu_seconds_total{job="otel-collector"}[5m])

Fix: Scale collectors horizontally

kubectl scale deployment otel-collector-gateway --replicas=5 -n observability

Production Checklist

Scaling Benchmarks

From my production deployment:

Metric

100 req/s

1,000 req/s

10,000 req/s

DaemonSet Collectors

1 per node

Gateway Collectors

Jaeger Collectors

Elasticsearch Nodes

Monthly Cost

$50

$200

$800

Data Generated

13 GB/day

130 GB/day

1.3 TB/day

What's Next

You've completed the core OpenTelemetry 101 series! Continue to Multi-Backend Integration for advanced topics on integrating with cloud providers and commercial observability platforms.

Previous: ← Security Best Practices | Next: Multi-Backend Integration →

Production-ready observability is a marathon, not a sprint.

PreviousSecurity Best Practices NextMulti-Backend Integration

Last updated 11 hours ago

hashtagFrom 10 to 10,000 Requests Per Second

hashtagArchitecture Evolution

hashtagPhase 1: Simple (0-100 req/s)

hashtagPhase 2: Distributed (100-1,000 req/s)

hashtagPhase 3: Scaled (1,000-10,000+ req/s)

hashtagProduction Kubernetes Deployment

hashtagCollector DaemonSet (Sidecar Pattern)

hashtagGateway Collector Deployment

hashtagDaemonSet Configuration

hashtagGateway Configuration

hashtagJaeger High Availability

hashtagElasticsearch Backend

hashtagMonitoring the Monitoring System

hashtagCollector Metrics

hashtagApplication Configuration

hashtagCost Optimization

hashtagStorage Costs

hashtagRunbook: Common Issues

hashtagIssue 1: Export Queue Full

hashtagIssue 2: Jaeger Storage Full

hashtagIssue 3: High Collector CPU

hashtagProduction Checklist

hashtagScaling Benchmarks

hashtagWhat's Next

From 10 to 10,000 Requests Per Second

Architecture Evolution

Phase 1: Simple (0-100 req/s)

Phase 2: Distributed (100-1,000 req/s)

Phase 3: Scaled (1,000-10,000+ req/s)

Production Kubernetes Deployment

Collector DaemonSet (Sidecar Pattern)

Gateway Collector Deployment

DaemonSet Configuration

Gateway Configuration

Jaeger High Availability

Elasticsearch Backend

Monitoring the Monitoring System

Collector Metrics

Application Configuration

Cost Optimization

Storage Costs

Runbook: Common Issues

Issue 1: Export Queue Full

Issue 2: Jaeger Storage Full

Issue 3: High Collector CPU

Production Checklist

Scaling Benchmarks

What's Next