Real-World Project - Building a Microservice

The Microservice That Survived Black Friday

It was 11:58 PM on Thanksgiving when our CTO sent the message: "We're expecting 10x traffic tomorrow. Are we ready?"

Our product catalog service was handling 500 requests per second normally. Black Friday would bring 5,000+ req/s. I spent that night stress-testing our Go microservice.

The Python version from last year had crashed at 800 req/s:

• Out of memory after 2 hours

• No graceful shutdown (interrupted orders)

• No health checks (load balancer sent traffic to dying pods)

• Manual restarts required

The Go rewrite was different:

// Health check endpoint
func (s *Server) healthHandler(w http.ResponseWriter, r *http.Request) {
    if !s.isHealthy() {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

// Graceful shutdown
func (s *Server) Shutdown(ctx context.Context) error {
    log.Info("Shutting down gracefully...")
    return s.httpServer.Shutdown(ctx)
}

// Metrics endpoint
func (s *Server) metricsHandler(w http.ResponseWriter, r *http.Request) {
    promhttp.Handler().ServeHTTP(w, r)
}

Black Friday results:

• Peak: 7,200 req/s (40% above estimate)

• Uptime: 100% (zero restarts)

• Memory: 180MB stable (no leaks)

• Response time: 23ms p99 (under load)

• Revenue: $1.2M processed without issue

That night taught me: production-ready isn't just working code. This article covers building microservices that survive real traffic.

---

Project Structure

product-service/
├── cmd/
│   └── server/
│       └── main.go              # Entry point
├── internal/
│   ├── config/
│   │   └── config.go            # Configuration
│   ├── product/
│   │   ├── handler.go           # HTTP handlers
│   │   ├── service.go           # Business logic
│   │   ├── repository.go        # Database access
│   │   └── model.go             # Data models
│   ├── middleware/
│   │   ├── logging.go
│   │   ├── metrics.go
│   │   └── recovery.go
│   └── server/
│       └── server.go            # HTTP server setup
├── pkg/
│   ├── database/
│   │   └── postgres.go
│   └── logger/
│       └── logger.go
├── deployments/
│   ├── Dockerfile
│   └── kubernetes/
│       ├── deployment.yaml
│       └── service.yaml
├── go.mod
├── go.sum
├── Makefile
└── README.md

---

Building the HTTP Server

Server Setup

internal/server/server.go:

package server

import (
    "context"
    "database/sql"
    "net/http"
    "time"
    
    "github.com/gorilla/mux"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    log "github.com/sirupsen/logrus"
    
    "product-service/internal/middleware"
    "product-service/internal/product"
)

type Server struct {
    httpServer *http.Server
    db         *sql.DB
}

func New(db *sql.DB, port string) *Server {
    router := mux.NewRouter()
    
    // Repository and service
    productRepo := product.NewRepository(db)
    productService := product.NewService(productRepo)
    productHandler := product.NewHandler(productService)
    
    // Middleware
    router.Use(middleware.Logging)
    router.Use(middleware.Metrics)
    router.Use(middleware.Recovery)
    
    // Health and metrics endpoints
    router.HandleFunc("/health", healthHandler).Methods("GET")
    router.HandleFunc("/ready", readyHandler(db)).Methods("GET")
    router.HandleFunc("/metrics", promhttp.Handler().ServeHTTP).Methods("GET")
    
    // API routes
    api := router.PathPrefix("/api/v1").Subrouter()
    api.HandleFunc("/products", productHandler.List).Methods("GET")
    api.HandleFunc("/products/{id}", productHandler.Get).Methods("GET")
    api.HandleFunc("/products", productHandler.Create).Methods("POST")
    api.HandleFunc("/products/{id}", productHandler.Update).Methods("PUT")
    api.HandleFunc("/products/{id}", productHandler.Delete).Methods("DELETE")
    
    httpServer := &http.Server{
        Addr:         ":" + port,
        Handler:      router,
        ReadTimeout:  15 * time.Second,
        WriteTimeout: 15 * time.Second,
        IdleTimeout:  60 * time.Second,
    }
    
    return &Server{
        httpServer: httpServer,
        db:         db,
    }
}

func (s *Server) Start() error {
    log.Infof("Starting server on %s", s.httpServer.Addr)
    return s.httpServer.ListenAndServe()
}

func (s *Server) Shutdown(ctx context.Context) error {
    log.Info("Shutting down server...")
    return s.httpServer.Shutdown(ctx)
}

func healthHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("OK"))
}

func readyHandler(db *sql.DB) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        if err := db.Ping(); err != nil {
            log.WithError(err).Error("Database ping failed")
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
        w.Write([]byte("Ready"))
    }
}

---

Graceful Shutdown

cmd/server/main.go:

package main

import (
    "context"
    "os"
    "os/signal"
    "syscall"
    "time"
    
    log "github.com/sirupsen/logrus"
    
    "product-service/internal/config"
    "product-service/internal/server"
    "product-service/pkg/database"
)

func main() {
    // Initialize logger
    log.SetFormatter(&log.JSONFormatter{})
    log.SetLevel(log.InfoLevel)
    
    // Load configuration
    cfg := config.Load()
    
    // Connect to database
    db, err := database.Connect(cfg.DatabaseURL)
    if err != nil {
        log.WithError(err).Fatal("Failed to connect to database")
    }
    defer db.Close()
    
    // Create server
    srv := server.New(db, cfg.Port)
    
    // Start server in goroutine
    go func() {
        if err := srv.Start(); err != nil && err != http.ErrServerClosed {
            log.WithError(err).Fatal("Server failed")
        }
    }()
    
    log.Info("Server started successfully")
    
    // Wait for interrupt signal
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
    
    log.Info("Received shutdown signal")
    
    // Graceful shutdown with timeout
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    
    if err := srv.Shutdown(ctx); err != nil {
        log.WithError(err).Fatal("Server forced to shutdown")
    }
    
    log.Info("Server stopped gracefully")
}

Why graceful shutdown matters:

• Finishes in-flight requests

• Closes database connections

• Flushes logs and metrics

• Prevents data corruption

---

Health Checks and Readiness Probes

Health Check

Simple liveness check - is the service running?

func healthHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("OK"))
}

Readiness Check

Deeper check - is the service ready to handle traffic?

type HealthChecker struct {
    db    *sql.DB
    cache *redis.Client
}

func (h *HealthChecker) Ready(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
    defer cancel()
    
    // Check database
    if err := h.db.PingContext(ctx); err != nil {
        log.WithError(err).Error("Database check failed")
        respondError(w, http.StatusServiceUnavailable, "Database unavailable")
        return
    }
    
    // Check cache
    if err := h.cache.Ping(ctx).Err(); err != nil {
        log.WithError(err).Error("Cache check failed")
        respondError(w, http.StatusServiceUnavailable, "Cache unavailable")
        return
    }
    
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]string{
        "status": "ready",
        "database": "ok",
        "cache": "ok",
    })
}

---

Metrics with Prometheus

Setup Prometheus

go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promauto
go get github.com/prometheus/client_golang/prometheus/promhttp

Define Metrics

internal/middleware/metrics.go:

package middleware

import (
    "net/http"
    "strconv"
    "time"
    
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    httpRequestsTotal = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "path", "status"},
    )
    
    httpRequestDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "http_request_duration_seconds",
            Help:    "HTTP request duration in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"method", "path"},
    )
    
    activeRequests = promauto.NewGauge(
        prometheus.GaugeOpts{
            Name: "http_requests_active",
            Help: "Number of active HTTP requests",
        },
    )
)

func Metrics(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        
        activeRequests.Inc()
        defer activeRequests.Dec()
        
        // Wrap ResponseWriter to capture status code
        rw := &responseWriter{w, http.StatusOK}
        
        next.ServeHTTP(rw, r)
        
        duration := time.Since(start).Seconds()
        
        httpRequestsTotal.WithLabelValues(
            r.Method,
            r.URL.Path,
            strconv.Itoa(rw.statusCode),
        ).Inc()
        
        httpRequestDuration.WithLabelValues(
            r.Method,
            r.URL.Path,
        ).Observe(duration)
    })
}

type responseWriter struct {
    http.ResponseWriter
    statusCode int
}

func (rw *responseWriter) WriteHeader(code int) {
    rw.statusCode = code
    rw.ResponseWriter.WriteHeader(code)
}

Expose Metrics

import "github.com/prometheus/client_golang/prometheus/promhttp"

router.HandleFunc("/metrics", promhttp.Handler().ServeHTTP).Methods("GET")

Access metrics:

curl http://localhost:8080/metrics

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",path="/api/v1/products",status="200"} 1523
http_requests_total{method="POST",path="/api/v1/products",status="201"} 234

# HELP http_request_duration_seconds HTTP request duration in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{method="GET",path="/api/v1/products",le="0.005"} 1234
http_request_duration_seconds_bucket{method="GET",path="/api/v1/products",le="0.01"} 1456

---

Structured Logging

pkg/logger/logger.go:

package logger

import (
    "os"
    
    log "github.com/sirupsen/logrus"
)

func Init(level string) {
    log.SetFormatter(&log.JSONFormatter{
        TimestampFormat: "2006-01-02T15:04:05.000Z",
    })
    
    log.SetOutput(os.Stdout)
    
    switch level {
    case "debug":
        log.SetLevel(log.DebugLevel)
    case "info":
        log.SetLevel(log.InfoLevel)
    case "warn":
        log.SetLevel(log.WarnLevel)
    case "error":
        log.SetLevel(log.ErrorLevel)
    default:
        log.SetLevel(log.InfoLevel)
    }
}

internal/middleware/logging.go:

package middleware

import (
    "net/http"
    "time"
    
    log "github.com/sirupsen/logrus"
)

func Logging(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        
        rw := &responseWriter{w, http.StatusOK}
        next.ServeHTTP(rw, r)
        
        log.WithFields(log.Fields{
            "method":     r.Method,
            "path":       r.URL.Path,
            "status":     rw.statusCode,
            "duration":   time.Since(start).Milliseconds(),
            "user_agent": r.UserAgent(),
            "remote_ip":  r.RemoteAddr,
        }).Info("HTTP request")
    })
}

Output:

{
  "level": "info",
  "msg": "HTTP request",
  "method": "GET",
  "path": "/api/v1/products",
  "status": 200,
  "duration": 23,
  "user_agent": "Mozilla/5.0",
  "remote_ip": "192.168.1.100",
  "time": "2024-01-15T14:23:45.123Z"
}

---

Docker Containerization

Multi-Stage Dockerfile

deployments/Dockerfile:

# Build stage
FROM golang:1.23-alpine AS builder

WORKDIR /app

# Copy go mod files
COPY go.mod go.sum ./
RUN go mod download

# Copy source code
COPY . .

# Build binary
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags="-s -w" -o server ./cmd/server

# Final stage
FROM alpine:latest

RUN apk --no-cache add ca-certificates

WORKDIR /root/

# Copy binary from builder
COPY --from=builder /app/server .

# Expose port
EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
    CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1

# Run
CMD ["./server"]

Build and run:

# Build image
docker build -t product-service:v1.0.0 -f deployments/Dockerfile .

# Run container
docker run -p 8080:8080 \
    -e DATABASE_URL=postgres://... \
    -e LOG_LEVEL=info \
    product-service:v1.0.0

# Check health
curl http://localhost:8080/health

Benefits of multi-stage build:

• Builder image: 800MB (has Go compiler)

• Final image: 15MB (only binary + Alpine)

• Faster deployment, less attack surface

---

Kubernetes Deployment

Deployment

deployments/kubernetes/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: product-service
  labels:
    app: product-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: product-service
  template:
    metadata:
      labels:
        app: product-service
    spec:
      containers:
      - name: product-service
        image: product-service:v1.0.0
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: product-service-secrets
              key: database-url
        - name: LOG_LEVEL
          value: "info"
        - name: PORT
          value: "8080"
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 30
          timeoutSeconds: 3
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 3

Service

deployments/kubernetes/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: product-service
  labels:
    app: product-service
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: product-service

Deploy to Kubernetes

# Create secrets
kubectl create secret generic product-service-secrets \
    --from-literal=database-url=postgres://...

# Deploy
kubectl apply -f deployments/kubernetes/deployment.yaml
kubectl apply -f deployments/kubernetes/service.yaml

# Check status
kubectl get pods -l app=product-service
kubectl get svc product-service

# Check logs
kubectl logs -l app=product-service --tail=100

# Port forward for testing
kubectl port-forward svc/product-service 8080:80
curl http://localhost:8080/health

---

Complete Working Example

internal/product/model.go:

package product

import "time"

type Product struct {
    ID          int       `json:"id" db:"id"`
    Name        string    `json:"name" db:"name"`
    Description string    `json:"description" db:"description"`
    Price       float64   `json:"price" db:"price"`
    Stock       int       `json:"stock" db:"stock"`
    CreatedAt   time.Time `json:"created_at" db:"created_at"`
    UpdatedAt   time.Time `json:"updated_at" db:"updated_at"`
}

internal/product/handler.go:

package product

import (
    "encoding/json"
    "net/http"
    "strconv"
    
    "github.com/gorilla/mux"
    log "github.com/sirupsen/logrus"
)

type Handler struct {
    service *Service
}

func NewHandler(service *Service) *Handler {
    return &Handler{service: service}
}

func (h *Handler) List(w http.ResponseWriter, r *http.Request) {
    products, err := h.service.List(r.Context())
    if err != nil {
        log.WithError(err).Error("Failed to list products")
        http.Error(w, "Internal server error", http.StatusInternalServerError)
        return
    }
    
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(products)
}

func (h *Handler) Get(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    id, err := strconv.Atoi(vars["id"])
    if err != nil {
        http.Error(w, "Invalid ID", http.StatusBadRequest)
        return
    }
    
    product, err := h.service.Get(r.Context(), id)
    if err != nil {
        log.WithError(err).Error("Failed to get product")
        http.Error(w, "Product not found", http.StatusNotFound)
        return
    }
    
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(product)
}

func (h *Handler) Create(w http.ResponseWriter, r *http.Request) {
    var product Product
    if err := json.NewDecoder(r.Body).Decode(&product); err != nil {
        http.Error(w, "Invalid request body", http.StatusBadRequest)
        return
    }
    
    created, err := h.service.Create(r.Context(), &product)
    if err != nil {
        log.WithError(err).Error("Failed to create product")
        http.Error(w, "Internal server error", http.StatusInternalServerError)
        return
    }
    
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusCreated)
    json.NewEncoder(w).Encode(created)
}

---

Makefile for Development

.PHONY: build run test docker-build docker-run k8s-deploy

VERSION := $(shell git describe --tags --always --dirty)
IMAGE := product-service:$(VERSION)

build:
	go build -o bin/server ./cmd/server

run:
	go run ./cmd/server

test:
	go test -v -race -cover ./...

lint:
	golangci-lint run

docker-build:
	docker build -t $(IMAGE) -f deployments/Dockerfile .

docker-run:
	docker run -p 8080:8080 $(IMAGE)

k8s-deploy:
	kubectl apply -f deployments/kubernetes/

clean:
	rm -rf bin/

---

Your Challenge

Build a complete TODO API microservice:

Requirements:
1. REST API with CRUD operations
2. PostgreSQL database
3. Health and readiness checks
4. Prometheus metrics
5. Structured logging
6. Graceful shutdown
7. Docker container (multi-stage)
8. Kubernetes deployment

Endpoints:
- GET /api/v1/todos
- GET /api/v1/todos/:id
- POST /api/v1/todos
- PUT /api/v1/todos/:id
- DELETE /api/v1/todos/:id

Bonus:
- Rate limiting middleware
- Authentication
- Database migrations
- Horizontal pod autoscaling

---

Key Takeaways

1. Graceful shutdown: Finish in-flight requests before stopping

2. Health checks: Liveness and readiness probes

3. Metrics: Prometheus for monitoring

4. Structured logging: JSON logs for observability

5. Multi-stage Docker: Small, secure images

6. Kubernetes: Production-grade orchestration

7. Resource limits: Prevent memory/CPU runaway

8. Middleware: Logging, metrics, recovery

---

What I Learned

That Black Friday microservice taught me production-ready means more than working code:

• Graceful shutdown prevented corrupted orders during deployments

• Health checks ensured traffic only went to healthy pods

• Metrics showed bottlenecks before they became incidents

• 7,200 req/s handled smoothly, 40% above estimates

• $1.2M revenue processed with 100% uptime

Coming from Python microservices that needed 4GB and struggled with concurrency, Go's efficiency was transformative. The service runs on 180MB RAM and handles more traffic than we ever expected.

Two years later, it's still the most reliable service we have. Zero production incidents. Zero manual restarts.

The confidence to sleep through Black Friday? Priceless.

---

Congratulations! 🎉

You've completed the Go 101 series! You've learned:

1. Go basics and why it's different

2. Functions, methods, and error handling

3. Data structures and memory management

4. Structs, interfaces, and composition

5. Error handling philosophy

6. Concurrency with goroutines and channels

7. Advanced concurrency patterns

8. Package management with modules

9. Testing and TDD

10. JSON and REST APIs

11. Database operations

12. Best practices and code organization

13. Building CLI tools

14. Building production microservices

Where to go next:

• Build real projects

• Contribute to open source

• Read "Effective Go" and "Go Proverbs"

• Join the Go community

• Keep learning and shipping

Welcome to the Go community. Now go build something amazing! 🚀