Kubernetes Operator Development 101 with Go

A hands-on series on building production-grade Kubernetes Operators using Go and kubebuilder. Every article is built around a real operator I built: appstack-operator — a tool that manages application stacks as a first-class Kubernetes resource.

About This Series

I started writing Kubernetes Operators after spending months writing Helm charts that were increasingly difficult to reason about. Every time application deployment needed custom logic — health-based rollback, automatic HPA tuning based on CI labels, or coordinated multi-resource rollouts — the answer was "write a bash script" or "add a CronJob". That's when I reached for the Operator pattern.

This series documents what I actually needed to know: the Go types, the reconcile loop, how to test without a real cluster, and how to deploy safely. No imaginary companies or contrived scenarios — just the code and the reasoning behind it.

All examples in this series come from my open-source project: appstack-operator — a Kubernetes operator that manages AppStack custom resources, coordinating Deployments, Services, and HPAs as a single lifecycle unit.

---

What You'll Build

An operator that introduces a single CRD:

apiVersion: apps.htunn.io/v1alpha1
kind: AppStack
metadata:
  name: api-service
  namespace: production
spec:
  image: ghcr.io/htunn/api-service:v1.2.0
  port: 8080
  replicas: 3
  env:
    - name: LOG_LEVEL
      value: info
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    cpuTargetPercent: 60

Applying this resource makes the operator:

1. Create or reconcile a Deployment

2. Create or reconcile a Service (ClusterIP)

3. Create or reconcile an HorizontalPodAutoscaler (if autoscaling.enabled)

4. Update .status.conditions to reflect the current health

5. Re-reconcile when any of the managed resources drift

---

Who This Series Is For

• Go developers who understand the Kubernetes object model and want to extend it

• Platform engineers tired of wrapping logic around plain Helm charts

• SREs looking to automate operational runbooks as code

• Anyone who has asked "why can't I just define this whole stack as one thing?"

---

Prerequisites

• Kubernetes basics: You understand Deployments, Services, and Pods

• Go fundamentals: Structs, interfaces, goroutines, error handling

• kubectl: Comfortable with kubectl get, apply, describe

• Local cluster: kind or k3d for development

---

Series Structure

Phase 1: Concepts and Setup

1. Introduction to Kubernetes Operators

   • The Controller pattern and how it differs from jobs/scripts

   • When you should build an operator (and when you shouldn't)

   • How kubebuilder, controller-runtime, and the Kubernetes API fit together

2. Project Setup with kubebuilder

   • Installing the kubebuilder CLI

   • Scaffolding the appstack-operator project

   • Understanding the generated project structure

   • Running the operator locally against a kind cluster

Phase 2: Core Implementation

1. CRD Design and Go API Types

   • Defining the AppStack spec and status in Go

   • kubebuilder validation markers

   • Generating CRD manifests with make manifests

   • Designing a status model with metav1.Condition

2. The Controller and Reconcile Loop

   • The reconcile loop in detail

   • Fetching the CR and handling NotFound

   • Creating and updating owned resources (Deployment, Service, HPA)

   • ctrl.Result — requeue strategies explained

3. Status, Events, and Observability

   • Writing status conditions correctly

   • Recording Kubernetes Events from the controller

   • Structured logging with logr

   • Exposing Prometheus metrics with controller-runtime

Phase 3: Testing and Deployment

1. Testing Operators with envtest

   • What envtest is and why you need it

   • Writing controller integration tests with Ginkgo/Gomega

   • Testing reconcile paths: create, update, delete

   • Faking the Kubernetes API in unit tests

2. RBAC, Deployment, and Production Hardening

   • RBAC markers and generated role manifests

   • Deploying the operator with make deploy

   • Leader election for HA operators

   • Resource limits, liveness probes, and security context

---

Toolchain Versions

All code in this series is tested against:

Tool	Version
Go	1.22
kubebuilder	v3.14
controller-runtime	v0.17
Kubernetes API	v0.29
kind	v0.22

---

Repository

Full source code: github.com/Htunn/appstack-operator

Each article maps to a tagged commit so you can follow along at any point in the series.