> For the complete documentation index, see [llms.txt](https://blog.htunnthuthu.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://blog.htunnthuthu.com/devops-and-sre/opentelemetry-101.md).

# OpenTelemetry 101

## My OpenTelemetry Journey

When I first started building distributed systems, debugging production issues felt like searching for a needle in a haystack. Logs were scattered, there was no visibility into request flows across services, and understanding system performance required manual correlation of disparate data sources. I knew I needed comprehensive observability, but vendor-specific solutions created lock-in, and piecing together different tools meant learning multiple instrumentation approaches.

That's when I discovered OpenTelemetry. What started as frustration with fragmented observability evolved into a deep appreciation for unified telemetry collection. Over the years, I've instrumented microservices in TypeScript, traced requests across distributed systems, and built custom exporters to fit specific backend needs. Every insight in this series comes from production implementations where proper observability made the difference between rapid incident resolution and prolonged outages.

This isn't just another OpenTelemetry tutorial - it's a comprehensive journey from basic instrumentation to production-grade observability strategies, all using TypeScript and Node.js as the foundation.

## What is OpenTelemetry?

OpenTelemetry is a vendor-neutral, open-source observability framework for generating, collecting, and exporting telemetry data (traces, metrics, and logs). It's a CNCF project born from the merger of OpenTracing and OpenCensus, combining the strengths of both to create a unified standard.

**Key Principles:**

* **You own your data**: No vendor lock-in, freedom to switch backends
* **Single set of APIs**: Learn once, use everywhere regardless of language or backend
* **Standardized instrumentation**: Consistent approach across your entire stack

## What You'll Master

This series takes you from zero to production-ready OpenTelemetry implementation:

### **Phase 1: Foundations (Week 1-2)**

1. **OpenTelemetry fundamentals** - Understanding traces, metrics, logs, and the OTel architecture
2. **TypeScript setup** - Instrumenting your first Node.js/TypeScript application
3. **Automatic instrumentation** - Leveraging auto-instrumentation for Express, HTTP, and databases

### **Phase 2: Manual Instrumentation (Week 3-4)**

4. **Custom spans and attributes** - Creating detailed traces for business logic
5. **Metrics implementation** - Counters, gauges, histograms for application monitoring
6. **Context propagation** - Distributed tracing across microservices

### **Phase 3: Advanced Patterns (Week 5-6)**

7. **Sampling strategies** - Managing telemetry volume in high-traffic systems
8. **Resource detection** - Automatic service identification and environment metadata
9. **Custom exporters** - Sending telemetry to multiple backends

### **Phase 4: Production Deployment (Week 7-8)**

10. **OpenTelemetry Collector** - Centralized telemetry pipeline management
11. **Performance optimization** - Minimizing instrumentation overhead
12. **Security and compliance** - Handling sensitive data in telemetry

### **Phase 5: Enterprise Observability (Week 9-10)**

13. **Multi-backend strategies** - Sending data to Prometheus, Jaeger, and cloud platforms
14. **Alerting and SLOs** - Building observability-driven reliability
15. **Production best practices** - Lessons from running OTel at scale

## Why This Matters

Modern applications are distributed by default. A single user request might flow through multiple services, databases, caches, and external APIs. Without proper observability:

* **Debugging is blind**: You can't see where requests slow down or fail
* **Performance optimization is guesswork**: You don't know which code paths need improvement
* **Incidents take longer to resolve**: No visibility means longer MTTR
* **Capacity planning is reactive**: You discover bottlenecks when users complain

OpenTelemetry solves these problems by providing:

✅ **Distributed tracing**: Follow requests across your entire system\
✅ **Metrics collection**: Track performance, errors, and resource usage\
✅ **Structured logs**: Correlate logs with traces and spans\
✅ **Vendor neutrality**: Switch backends without changing instrumentation\
✅ **Automatic instrumentation**: Get started quickly with minimal code changes\
✅ **Extensibility**: Build custom instrumentation for your specific needs

## The Three Pillars of Telemetry

OpenTelemetry revolves around three core signals:

### **1. Traces**

Distributed traces show the journey of a request through your system. Each trace contains spans representing units of work, forming a parent-child relationship that visualizes the entire request lifecycle.

**Use cases:**

* Identifying slow database queries
* Finding bottlenecks in API calls
* Understanding request flow in microservices
* Debugging distributed transactions

### **2. Metrics**

Metrics are numeric measurements over time - counters, gauges, and histograms that quantify system behavior.

**Use cases:**

* Monitoring request rates and error rates
* Tracking memory and CPU usage
* Measuring business KPIs (orders processed, revenue)
* Setting up alerts and SLOs

### **3. Logs**

Structured log messages that can be correlated with traces and metrics using trace IDs and span IDs.

**Use cases:**

* Detailed error investigation
* Auditing and compliance
* Business event tracking
* Development debugging

## OpenTelemetry Architecture

The OpenTelemetry ecosystem consists of several key components:

```
┌─────────────────────────────────────────────────────────────┐
│                      Your Application                        │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   OTel API   │  │  OTel SDK    │  │ Auto-Instr.  │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────┬───────────────────────────────────────────┘
                  │ Telemetry Data
                  ▼
         ┌────────────────────┐
         │  OTel Collector    │ (Optional)
         │  - Receive         │
         │  - Process         │
         │  - Export          │
         └────────┬───────────┘
                  │
         ┌────────┴────────┐
         │                 │
    ┌────▼─────┐     ┌────▼─────┐
    │ Jaeger   │     │Prometheus│
    │ (Traces) │     │(Metrics) │
    └──────────┘     └──────────┘
```

**Components:**

* **API**: Language-specific interfaces for creating telemetry
* **SDK**: Implementation of the API with configuration and export capabilities
* **Auto-instrumentation**: Automatic telemetry for popular libraries
* **Collector**: Vendor-agnostic telemetry pipeline (optional but recommended for production)
* **Exporters**: Send data to observability backends

## Real-World Implementation Preview

Throughout this series, I'll build a production-ready TypeScript microservice with comprehensive observability:

**Project: E-commerce Order Service**

* Express.js REST API with TypeScript
* PostgreSQL database with TypeORM
* Redis caching layer
* External payment API integration
* Background job processing
* Full OpenTelemetry instrumentation

You'll see:

* Automatic instrumentation for HTTP, database, and Redis
* Custom spans for business logic (order validation, payment processing)
* Metrics for order rates, payment success/failure, inventory levels
* Distributed tracing across service boundaries
* Performance optimization using telemetry data
* Production deployment with the OTel Collector

## What Makes This Different

I'm not building contrived examples or theoretical scenarios. Every pattern in this series comes from:

* **Production experience**: Instrumenting real microservices handling millions of requests
* **Actual debugging stories**: Times when proper observability saved hours of investigation
* **Performance lessons**: Optimizing instrumentation overhead in high-throughput systems
* **Migration experiences**: Moving from vendor-specific tools to OpenTelemetry
* **Team collaboration**: Building observability practices that scale across engineering teams

## Prerequisites

To get the most from this series, you should have:

**Required:**

* Solid TypeScript/JavaScript knowledge
* Node.js development experience
* Understanding of async/await and promises
* Basic familiarity with Express.js or similar frameworks

**Helpful but not required:**

* Experience with distributed systems
* Knowledge of observability concepts
* Exposure to monitoring tools (Prometheus, Grafana, Jaeger)
* Docker and containerization basics

## Learning Path

Each article in this series builds on the previous ones:

1. [**OpenTelemetry Fundamentals**](/devops-and-sre/opentelemetry-101/opentelemetry-101-fundamentals.md) - Core concepts, signals, and architecture
2. [**Getting Started with TypeScript**](/devops-and-sre/opentelemetry-101/opentelemetry-101-typescript-setup.md) - First instrumented application
3. [**Automatic Instrumentation**](/devops-and-sre/opentelemetry-101/opentelemetry-101-auto-instrumentation.md) - Leverage community libraries
4. [**Manual Instrumentation Deep Dive**](/devops-and-sre/opentelemetry-101/opentelemetry-101-manual-instrumentation.md) - Custom spans and attributes
5. [**Metrics Collection**](/devops-and-sre/opentelemetry-101/opentelemetry-101-metrics.md) - Counters, gauges, and histograms
6. [**Distributed Tracing**](/devops-and-sre/opentelemetry-101/opentelemetry-101-distributed-tracing.md) - Context propagation across services
7. [**Sampling Strategies**](/devops-and-sre/opentelemetry-101/opentelemetry-101-sampling.md) - Managing telemetry volume
8. [**Resource Detection**](https://github.com/Htunn/git-book/blob/main/automation/opentelemetry-101/opentelemetry-101-resource-detection.md) - Service identification and metadata
9. [**Custom Exporters**](https://github.com/Htunn/git-book/blob/main/automation/opentelemetry-101/opentelemetry-101-custom-exporters.md) - Multi-backend telemetry
10. [**OpenTelemetry Collector**](/devops-and-sre/opentelemetry-101/opentelemetry-101-collector.md) - Centralized pipeline management
11. [**Performance Optimization**](/devops-and-sre/opentelemetry-101/opentelemetry-101-performance.md) - Minimizing overhead
12. [**Security Best Practices**](/devops-and-sre/opentelemetry-101/opentelemetry-101-security.md) - Protecting sensitive data
13. [**Production Deployment**](https://github.com/Htunn/git-book/blob/main/automation/opentelemetry-101/opentelemetry-101-production.md) - Running OTel at scale
14. [**Multi-Backend Integration**](/devops-and-sre/opentelemetry-101/opentelemetry-101-multi-backend.md) - Jaeger, Prometheus, cloud platforms
15. [**Observability-Driven Development**](https://github.com/Htunn/git-book/blob/main/automation/opentelemetry-101/opentelemetry-101-observability-driven-development.md) - Building observable systems

## Quick Start Example

Here's a taste of what you'll learn - a simple Express TypeScript app with OpenTelemetry:

```typescript
// instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  traceExporter: new ConsoleSpanExporter(),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// app.ts
import express from 'express';

const app = express();

app.get('/api/orders/:id', async (req, res) => {
  // Automatically traced!
  const order = await db.orders.findOne({ id: req.params.id });
  res.json(order);
});

app.listen(3000);
```

Run it with:

```bash
npx tsx --import ./instrumentation.ts app.ts
```

Every HTTP request, database query, and Redis operation is automatically traced with zero code changes!

## What You'll Build

By the end of this series, you'll have:

✅ A fully instrumented TypeScript microservice\
✅ Comprehensive distributed tracing across services\
✅ Custom metrics for business and technical KPIs\
✅ Production-ready OpenTelemetry Collector configuration\
✅ Multi-backend observability (Jaeger + Prometheus + Cloud)\
✅ Performance-optimized instrumentation\
✅ Security-compliant telemetry handling\
✅ Alerting and SLO strategies\
✅ Team-ready observability practices

## Community and Resources

OpenTelemetry has a vibrant community:

* **Official Docs**: [opentelemetry.io/docs](https://opentelemetry.io/docs/)
* **GitHub**: [github.com/open-telemetry](https://github.com/open-telemetry)
* **CNCF Slack**: #opentelemetry channel
* **Registry**: [opentelemetry.io/registry](https://opentelemetry.io/ecosystem/registry/)
* **YouTube**: [OTel Community Channel](https://www.youtube.com/@otel-official)

## Ready to Begin?

Observability isn't optional anymore - it's the foundation of reliable software. Whether you're debugging a production incident, optimizing performance, or building new features, proper telemetry makes everything easier.

Let's start with [OpenTelemetry Fundamentals](/devops-and-sre/opentelemetry-101/opentelemetry-101-fundamentals.md) to understand the core concepts and architecture.

***

*This series reflects years of production OpenTelemetry experience. Every pattern, optimization, and best practice comes from real systems serving real users. Let's build observable, reliable software together.*