Security Best Practices

The PII Leak That Never Happened

During a security audit, our team reviewed traces in Jaeger and found:

{
  "traceId": "abc123...",
  "spans": [{
    "attributes": {
      "user.email": "[email protected]",
      "user.ssn": "123-45-6789",
      "credit_card.number": "4532-1234-5678-9010",
      "user.password": "MyP@ssw0rd!"
    }
  }]
}

We were logging PII (Personally Identifiable Information) in production traces!

This was a GDPR nightmare waiting to happen. Anyone with Jaeger access could see:

• Email addresses

• Social security numbers

• Credit card numbers

• Passwords (!)

The fix? Sanitize all telemetry data before export.

Rule #1: Never Log Sensitive Data

What Counts as Sensitive?

• PII: Email, phone, name, address, SSN, passport

• Credentials: Passwords, API keys, tokens, secrets

• Financial: Credit cards, bank accounts, transaction details

• Health: Medical records, diagnoses, prescriptions

• Proprietary: Trade secrets, internal algorithms, pricing

Automatic Redaction

import { ReadableSpan, SpanExporter } from '@opentelemetry/sdk-trace-base';
import { ExportResult, ExportResultCode } from '@opentelemetry/core';
import * as crypto from 'crypto';

const SENSITIVE_KEYS = [
  'password',
  'token',
  'api_key',
  'apiKey',
  'secret',
  'authorization',
  'cookie',
  'credit_card',
  'creditCard',
  'ssn',
  'social_security',
];

const PII_KEYS = [
  'email',
  'phone',
  'phoneNumber',
  'address',
  'ip_address',
  'ipAddress',
];

function hashValue(value: string): string {
  return crypto.createHash('sha256').update(value).digest('hex').substring(0, 16);
}

function shouldRedact(key: string): 'delete' | 'hash' | 'keep' {
  const lowerKey = key.toLowerCase();
  
  // Completely remove secrets
  if (SENSITIVE_KEYS.some(k => lowerKey.includes(k))) {
    return 'delete';
  }
  
  // Hash PII (preserves uniqueness for correlation)
  if (PII_KEYS.some(k => lowerKey.includes(k))) {
    return 'hash';
  }
  
  return 'keep';
}

export class SanitizingExporter implements SpanExporter {
  private wrapped: SpanExporter;
  
  constructor(exporter: SpanExporter) {
    this.wrapped = exporter;
  }
  
  private sanitizeAttributes(attributes: any): any {
    const sanitized: any = {};
    
    for (const [key, value] of Object.entries(attributes)) {
      const action = shouldRedact(key);
      
      switch (action) {
        case 'delete':
          // Don't include this attribute
          break;
        
        case 'hash':
          // Hash the value
          if (typeof value === 'string') {
            sanitized[key] = hashValue(value);
          }
          break;
        
        case 'keep':
        default:
          sanitized[key] = value;
      }
    }
    
    return sanitized;
  }
  
  async export(
    spans: ReadableSpan[],
    resultCallback: (result: ExportResult) => void
  ): Promise<void> {
    const sanitized = spans.map(span => ({
      ...span,
      attributes: this.sanitizeAttributes(span.attributes),
    }));
    
    return this.wrapped.export(sanitized as ReadableSpan[], resultCallback);
  }
  
  async shutdown(): Promise<void> {
    return this.wrapped.shutdown();
  }
  
  async forceFlush(): Promise<void> {
    return this.wrapped.forceFlush();
  }
}

// Usage
import { NodeSDK } from '@opentelemetry/sdk-node';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const sdk = new NodeSDK({
  spanProcessors: [
    new BatchSpanProcessor(
      new SanitizingExporter(
        new OTLPTraceExporter({
          url: 'http://localhost:4318/v1/traces',
        })
      )
    ),
  ],
});

sdk.start();

Result

Before sanitization:

{
  "user.email": "[email protected]",
  "user.password": "MyP@ssw0rd!",
  "credit_card.number": "4532-1234-5678-9010"
}

After sanitization:

{
  "user.email": "a1b2c3d4e5f6g7h8",  // Hashed (correlatable)
  // password: removed
  // credit_card.number: removed
}

Sanitizing HTTP Headers

HTTP headers often contain sensitive data:

import { context, propagation } from '@opentelemetry/api';
import axios from 'axios';

const SENSITIVE_HEADERS = [
  'authorization',
  'cookie',
  'x-api-key',
  'x-auth-token',
];

function sanitizeHeaders(headers: any): any {
  const sanitized = { ...headers };
  
  for (const header of SENSITIVE_HEADERS) {
    if (sanitized[header]) {
      delete sanitized[header];
    }
    
    // Also check lowercase
    if (sanitized[header.toLowerCase()]) {
      delete sanitized[header.toLowerCase()];
    }
  }
  
  return sanitized;
}

// Custom HTTP client wrapper
async function secureHttpPost(url: string, data: any, headers: any = {}) {
  return await tracer.startActiveSpan('http.post', async (span) => {
    try {
      span.setAttribute('http.url', url);
      span.setAttribute('http.method', 'POST');
      
      // DON'T log full headers
      // span.setAttribute('http.headers', headers); // ❌
      
      // Only log safe headers
      span.setAttribute('http.content_type', headers['content-type'] || 'application/json');
      
      // Propagate trace context
      const traceHeaders = propagation.inject(context.active(), {});
      
      const response = await axios.post(url, data, {
        headers: { ...headers, ...traceHeaders },
      });
      
      span.setAttribute('http.status_code', response.status);
      
      return response;
    } finally {
      span.end();
    }
  });
}

Collector-Level Sanitization

Use the OpenTelemetry Collector to sanitize at the infrastructure level:

processors:
  # Remove sensitive attributes
  attributes/redact:
    actions:
      # Delete secrets
      - key: password
        action: delete
      - key: api_key
        action: delete
      - key: apiKey
        action: delete
      - key: token
        action: delete
      - key: credit_card.number
        action: delete
      
      # Delete sensitive headers
      - key: http.request.header.authorization
        action: delete
      - key: http.request.header.cookie
        action: delete
      - key: http.response.header.set-cookie
        action: delete
      
      # Hash PII
      - key: user.email
        action: hash
      - key: user.phone
        action: hash
      - key: user.ip_address
        action: hash
  
  # Redact sensitive data in span names
  transform:
    trace_statements:
      - context: span
        statements:
          # Remove email from span names
          - replace_pattern(name, "email=([^&]+)", "email=REDACTED")
          # Remove tokens from span names
          - replace_pattern(name, "token=([^&]+)", "token=REDACTED")

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes/redact, transform, batch]
      exporters: [otlp/jaeger]

Securing Exporter Endpoints

TLS for Data in Transit

Bad: Plain HTTP

// ❌ Unencrypted
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
});

Good: HTTPS with TLS

import * as fs from 'fs';

// ✅ Encrypted
const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'https://otel-collector:4318/v1/traces',
    credentials: {
      ca: fs.readFileSync('./ca-cert.pem'),
      cert: fs.readFileSync('./client-cert.pem'),
      key: fs.readFileSync('./client-key.pem'),
    },
  }),
});

Authentication

API Key authentication:

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'https://otel-collector:4318/v1/traces',
    headers: {
      'x-api-key': process.env.OTEL_API_KEY!,
    },
  }),
});

OAuth2 Bearer Token:

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'https://otel-collector:4318/v1/traces',
    headers: {
      'authorization': `Bearer ${process.env.OTEL_ACCESS_TOKEN!}`,
    },
  }),
});

Access Control for Telemetry Data

Jaeger RBAC

Restrict access to traces:

# Jaeger with OAuth2
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
spec:
  template:
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/all-in-one:latest
        env:
        - name: QUERY_BASE_PATH
          value: /jaeger
        - name: ADMIN_HTTP_HOST_PORT
          value: :16687
        ports:
        - containerPort: 16686
          name: ui
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger-oauth2-proxy
spec:
  selector:
    app: oauth2-proxy
  ports:
  - port: 4180
    targetPort: 4180
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: oauth2-proxy
spec:
  template:
    spec:
      containers:
      - name: oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:latest
        args:
        - --provider=google
        - --email-domain=example.com  # Only allow company emails
        - --upstream=http://jaeger:16686
        - --http-address=0.0.0.0:4180
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: oauth2-secret
              key: client-id
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth2-secret
              key: client-secret

Data Retention and Deletion

Jaeger Retention Policies

# Limit data retention for compliance
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-collector
spec:
  template:
    spec:
      containers:
      - name: jaeger
        image: jaegertracing/jaeger-collector:latest
        env:
        # Delete traces after 7 days
        - name: ES_INDEX_DATE_SEPARATOR
          value: "."
        - name: ES_INDEX_ROLLOVER_FREQUENCY_SPANS
          value: "day"
        command:
        - "/go/bin/collector-linux"
        - "--es.num-shards=5"
        - "--es.num-replicas=1"
        - "--es.max-span-age=168h"  # 7 days

Automated Data Deletion

// Script to delete old traces
import { Client } from '@elastic/elasticsearch';

const client = new Client({
  node: 'http://elasticsearch:9200'
});

async function deleteOldTraces(daysToKeep: number) {
  const cutoffDate = new Date();
  cutoffDate.setDate(cutoffDate.getDate() - daysToKeep);
  
  const indices = await client.cat.indices({ format: 'json' });
  
  for (const index of indices.body) {
    if (index.index.startsWith('jaeger-span-')) {
      const indexDate = new Date(index.index.split('-').pop()!);
      
      if (indexDate < cutoffDate) {
        console.log(`Deleting old index: ${index.index}`);
        await client.indices.delete({ index: index.index });
      }
    }
  }
}

// Run daily
setInterval(() => {
  deleteOldTraces(7);  // Keep 7 days
}, 24 * 60 * 60 * 1000);

Compliance Considerations

GDPR Right to Deletion

Implement trace deletion by user ID:

import { Client } from '@elastic/elasticsearch';

async function deleteUserTraces(userId: string) {
  const client = new Client({ node: 'http://elasticsearch:9200' });
  
  await client.deleteByQuery({
    index: 'jaeger-span-*',
    body: {
      query: {
        term: {
          'tags.user.id': userId
        }
      }
    }
  });
  
  console.log(`Deleted all traces for user ${userId}`);
}

SOC 2 Audit Trail

Log access to telemetry:

import { createLogger, transports } from 'winston';

const auditLogger = createLogger({
  transports: [
    new transports.File({ filename: '/var/log/telemetry-audit.log' })
  ]
});

// Middleware for Jaeger UI
app.use('/jaeger', (req, res, next) => {
  auditLogger.info({
    event: 'telemetry_access',
    user: req.user.email,
    ip: req.ip,
    path: req.path,
    timestamp: new Date().toISOString()
  });
  next();
});

Environment-Specific Sanitization

Development: Less strict, more data for debugging

const isDevelopment = process.env.NODE_ENV === 'development';

function createExporter() {
  const baseExporter = new OTLPTraceExporter({
    url: process.env.OTEL_ENDPOINT,
  });
  
  // Only sanitize in production
  if (isDevelopment) {
    return baseExporter;
  } else {
    return new SanitizingExporter(baseExporter);
  }
}

const sdk = new NodeSDK({
  spanProcessors: [
    new BatchSpanProcessor(createExporter())
  ],
});

Security Checklist

• Sanitize PII before export

• Never log passwords, tokens, or secrets

• Use HTTPS/TLS for exporter endpoints

• Authenticate exporter connections

• Implement RBAC for Jaeger/observability tools

• Set data retention policies (7-30 days)

• Implement GDPR right-to-deletion

• Audit access to telemetry data

• Encrypt telemetry data at rest

• Review traces for sensitive data regularly

Real Security Incident

What happened: Developer accidentally logged entire request bodies, including credit card numbers.

Detection: Security scan of Jaeger found credit_card.number attribute.

Impact: 15,000 credit card numbers exposed in traces.

Response:

1. Immediately deleted all traces (7 days retention)

2. Deployed sanitizing exporter

3. Implemented Collector-level redaction

4. Added automated scanning for PII in traces

5. Notified affected customers (regulatory requirement)

Lesson: Defense in depth - sanitize at app, collector, and storage levels.

What's Next

Continue to Production Deployment to learn:

• Deploying at scale

• High availability setup

• Monitoring the monitoring system

• Runbooks and incident response

---

Previous: ← Performance Optimization | Next: Production Deployment →

Security first, observability second.