Testing Operators with envtest

Table of Contents

1. Introduction

2. What Is envtest?

3. Setting Up the Test Suite

4. Writing Controller Integration Tests

5. Testing Reconcile Paths

6. Unit Testing the Build Functions

7. Running Tests in CI

8. What I Test and What I Don't

---

Introduction

Testing a Kubernetes controller against a mock is almost useless — the interesting behavior is in how the controller interacts with the Kubernetes API. You need real watch semantics, real owner reference garbage collection, and real status subresource behavior.

envtest gives you exactly that: a real kube-apiserver and etcd binary running locally, no cluster needed. Your tests run the actual controller against the actual API machinery. This is what the controller-runtime team uses to test controller-runtime itself.

This article covers the test structure from appstack-operator, including both controller integration tests (Ginkgo/Gomega) and pure unit tests for the build functions.

---

What Is envtest?

envtest (from sigs.k8s.io/controller-runtime/pkg/envtest) downloads and runs local kube-apiserver and etcd binaries. Your test process:

1. Starts a real API server and etcd

2. Registers your CRDs

3. Runs your controllers against it

4. Cleans up when tests finish

The binaries are managed by setup-envtest, a tool from the controller-runtime project:

# Install setup-envtest
go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest

# Download binaries for your platform (uses Kubernetes 1.29)
setup-envtest use 1.29 --bin-dir /usr/local/kubebuilder/bin

# Set the environment variable (required by envtest)
export KUBEBUILDER_ASSETS=$(setup-envtest use 1.29 -p path)

The Makefile generated by kubebuilder handles this automatically when you run make test.

---

Setting Up the Test Suite

kubebuilder generates internal/controller/suite_test.go. The scaffold is minimal — here's what a complete setup looks like for appstack-operator:

package controller_test

import (
    "context"
    "path/filepath"
    "testing"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"

    appsv1 "k8s.io/api/apps/v1"
    autoscalingv2 "k8s.io/api/autoscaling/v2"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/runtime"
    clientgoscheme "k8s.io/client-go/kubernetes/scheme"
    "k8s.io/client-go/rest"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    logf "sigs.k8s.io/controller-runtime/pkg/log"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"

    appsv1alpha1 "github.com/htunn/appstack-operator/api/v1alpha1"
    "github.com/htunn/appstack-operator/internal/controller"
)

var (
    cfg       *rest.Config
    k8sClient client.Client
    testEnv   *envtest.Environment
    ctx       context.Context
    cancel    context.CancelFunc
    scheme    = runtime.NewScheme()
)

func TestControllers(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Controller Suite")
}

var _ = BeforeSuite(func() {
    logf.SetLogger(zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)))

    ctx, cancel = context.WithCancel(context.TODO())

    testEnv = &envtest.Environment{
        CRDDirectoryPaths: []string{
            filepath.Join("..", "..", "config", "crd", "bases"),
        },
        ErrorIfCRDPathMissing: true,
    }

    var err error
    cfg, err = testEnv.Start()
    Expect(err).NotTo(HaveOccurred())
    Expect(cfg).NotTo(BeNil())

    // Register schemes
    Expect(clientgoscheme.AddToScheme(scheme)).To(Succeed())
    Expect(appsv1alpha1.AddToScheme(scheme)).To(Succeed())

    k8sClient, err = client.New(cfg, client.Options{Scheme: scheme})
    Expect(err).NotTo(HaveOccurred())
    Expect(k8sClient).NotTo(BeNil())

    // Start the manager with the real environment
    mgr, err := ctrl.NewManager(cfg, ctrl.Options{
        Scheme: scheme,
    })
    Expect(err).NotTo(HaveOccurred())

    err = (&controller.AppStackReconciler{
        Client:   mgr.GetClient(),
        Scheme:   mgr.GetScheme(),
        Recorder: mgr.GetEventRecorderFor("appstack-controller"),
    }).SetupWithManager(mgr)
    Expect(err).NotTo(HaveOccurred())

    // Start the manager in a goroutine — it runs for the duration of the test suite
    go func() {
        defer GinkgoRecover()
        err = mgr.Start(ctx)
        Expect(err).NotTo(HaveOccurred())
    }()
})

var _ = AfterSuite(func() {
    cancel()
    Expect(testEnv.Stop()).To(Succeed())
})

Key points:

• testEnv.Start() launches kube-apiserver and etcd and returns a *rest.Config

• CRDDirectoryPaths points at your generated CRD YAML (the output of make manifests)

• The manager runs in a goroutine for the test suite's lifetime

• cancel() in AfterSuite stops the manager cleanly

---

Writing Controller Integration Tests

Integration tests live in internal/controller/appstack_controller_test.go:

package controller_test

import (
    "fmt"
    "time"

    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"

    appsv1 "k8s.io/api/apps/v1"
    autoscalingv2 "k8s.io/api/autoscaling/v2"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/types"
    "sigs.k8s.io/controller-runtime/pkg/client"

    appsv1alpha1 "github.com/htunn/appstack-operator/api/v1alpha1"
)

const (
    timeout  = 10 * time.Second
    interval = 100 * time.Millisecond
)

var _ = Describe("AppStack Controller", func() {
    Context("When creating an AppStack", func() {
        var appStack *appsv1alpha1.AppStack
        var namespace string

        BeforeEach(func() {
            // Use a unique namespace per test to avoid collisions
            namespace = fmt.Sprintf("test-%d", GinkgoRandomSeed())
            ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: namespace}}
            Expect(k8sClient.Create(ctx, ns)).To(Succeed())

            appStack = &appsv1alpha1.AppStack{
                ObjectMeta: metav1.ObjectMeta{
                    Name:      "test-stack",
                    Namespace: namespace,
                },
                Spec: appsv1alpha1.AppStackSpec{
                    Image:    "ghcr.io/htunn/api-service:v1.0.0",
                    Port:     8080,
                    Replicas: ptr(int32(2)),
                },
            }
            Expect(k8sClient.Create(ctx, appStack)).To(Succeed())
        })

        AfterEach(func() {
            Expect(k8sClient.Delete(ctx, appStack)).To(Succeed())
            ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: namespace}}
            Expect(k8sClient.Delete(ctx, ns)).To(Succeed())
        })

        It("should create a Deployment with the correct image and replicas", func() {
            deployment := &appsv1.Deployment{}
            Eventually(func() error {
                return k8sClient.Get(ctx, types.NamespacedName{
                    Name:      "test-stack",
                    Namespace: namespace,
                }, deployment)
            }, timeout, interval).Should(Succeed())

            Expect(deployment.Spec.Template.Spec.Containers).To(HaveLen(1))
            Expect(deployment.Spec.Template.Spec.Containers[0].Image).To(Equal("ghcr.io/htunn/api-service:v1.0.0"))
            Expect(*deployment.Spec.Replicas).To(Equal(int32(2)))
        })

        It("should create a Service targeting the correct port", func() {
            svc := &corev1.Service{}
            Eventually(func() error {
                return k8sClient.Get(ctx, types.NamespacedName{
                    Name:      "test-stack",
                    Namespace: namespace,
                }, svc)
            }, timeout, interval).Should(Succeed())

            Expect(svc.Spec.Ports).To(HaveLen(1))
            Expect(svc.Spec.Ports[0].TargetPort.IntVal).To(Equal(int32(8080)))
        })

        It("should set the AppStack phase to Pending initially", func() {
            // The deployed Deployment won't have ready replicas in envtest
            // (no actual pod scheduling), so phase should be Pending not Running
            Eventually(func() appsv1alpha1.AppStackPhase {
                updated := &appsv1alpha1.AppStack{}
                if err := k8sClient.Get(ctx, types.NamespacedName{
                    Name:      "test-stack",
                    Namespace: namespace,
                }, updated); err != nil {
                    return ""
                }
                return updated.Status.Phase
            }, timeout, interval).Should(Equal(appsv1alpha1.AppStackPhasePending))
        })

        It("should set the finalizer on the AppStack", func() {
            updated := &appsv1alpha1.AppStack{}
            Eventually(func() bool {
                if err := k8sClient.Get(ctx, types.NamespacedName{
                    Name:      "test-stack",
                    Namespace: namespace,
                }, updated); err != nil {
                    return false
                }
                for _, f := range updated.Finalizers {
                    if f == "apps.htunn.io/appstack-finalizer" {
                        return true
                    }
                }
                return false
            }, timeout, interval).Should(BeTrue())
        })
    })

    Context("When autoscaling is enabled", func() {
        var appStack *appsv1alpha1.AppStack
        var namespace string

        BeforeEach(func() {
            namespace = fmt.Sprintf("test-hpa-%d", GinkgoRandomSeed())
            ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: namespace}}
            Expect(k8sClient.Create(ctx, ns)).To(Succeed())

            appStack = &appsv1alpha1.AppStack{
                ObjectMeta: metav1.ObjectMeta{
                    Name:      "scaled-stack",
                    Namespace: namespace,
                },
                Spec: appsv1alpha1.AppStackSpec{
                    Image: "ghcr.io/htunn/api-service:v1.0.0",
                    Port:  8080,
                    Autoscaling: &appsv1alpha1.AutoscalingSpec{
                        Enabled:          true,
                        MinReplicas:      ptr(int32(2)),
                        MaxReplicas:      10,
                        CPUTargetPercent: 60,
                    },
                },
            }
            Expect(k8sClient.Create(ctx, appStack)).To(Succeed())
        })

        AfterEach(func() {
            Expect(k8sClient.Delete(ctx, appStack)).To(Succeed())
            ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: namespace}}
            Expect(k8sClient.Delete(ctx, ns)).To(Succeed())
        })

        It("should create an HPA with the correct target", func() {
            hpa := &autoscalingv2.HorizontalPodAutoscaler{}
            Eventually(func() error {
                return k8sClient.Get(ctx, types.NamespacedName{
                    Name:      "scaled-stack",
                    Namespace: namespace,
                }, hpa)
            }, timeout, interval).Should(Succeed())

            Expect(hpa.Spec.MaxReplicas).To(Equal(int32(10)))
            Expect(*hpa.Spec.MinReplicas).To(Equal(int32(2)))
            Expect(hpa.Spec.Metrics[0].Resource.Target.AverageUtilization).To(
                Equal(ptr(int32(60))),
            )
        })
    })

    Context("Image update", func() {
        var appStack *appsv1alpha1.AppStack
        var namespace string

        BeforeEach(func() {
            namespace = fmt.Sprintf("test-update-%d", GinkgoRandomSeed())
            ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: namespace}}
            Expect(k8sClient.Create(ctx, ns)).To(Succeed())

            appStack = &appsv1alpha1.AppStack{
                ObjectMeta: metav1.ObjectMeta{
                    Name:      "update-stack",
                    Namespace: namespace,
                },
                Spec: appsv1alpha1.AppStackSpec{
                    Image: "ghcr.io/htunn/api-service:v1.0.0",
                    Port:  8080,
                },
            }
            Expect(k8sClient.Create(ctx, appStack)).To(Succeed())

            // Wait for initial Deployment to be created
            deployment := &appsv1.Deployment{}
            Eventually(func() error {
                return k8sClient.Get(ctx, types.NamespacedName{
                    Name: "update-stack", Namespace: namespace,
                }, deployment)
            }, timeout, interval).Should(Succeed())
        })

        AfterEach(func() {
            _ = k8sClient.Delete(ctx, appStack)
            ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: namespace}}
            _ = k8sClient.Delete(ctx, ns)
        })

        It("should update the Deployment when the image changes", func() {
            // Update the image
            updated := &appsv1alpha1.AppStack{}
            Expect(k8sClient.Get(ctx, types.NamespacedName{
                Name: "update-stack", Namespace: namespace,
            }, updated)).To(Succeed())

            updated.Spec.Image = "ghcr.io/htunn/api-service:v2.0.0"
            Expect(k8sClient.Update(ctx, updated)).To(Succeed())

            // Verify Deployment reflects the new image
            deployment := &appsv1.Deployment{}
            Eventually(func() string {
                if err := k8sClient.Get(ctx, types.NamespacedName{
                    Name: "update-stack", Namespace: namespace,
                }, deployment); err != nil {
                    return ""
                }
                if len(deployment.Spec.Template.Spec.Containers) == 0 {
                    return ""
                }
                return deployment.Spec.Template.Spec.Containers[0].Image
            }, timeout, interval).Should(Equal("ghcr.io/htunn/api-service:v2.0.0"))
        })
    })
})

// ptr returns a pointer to a value — used for optional spec fields
func ptr[T any](v T) *T {
    return &v
}

---

Testing Reconcile Paths

Why Eventually Instead of Direct Assertions

Tests against a controller are asynchronous. After k8sClient.Create(), the controller's reconcile loop runs independently. You can't assert immediately after create:

// WRONG — the controller hasn't run yet
k8sClient.Create(ctx, appStack)
k8sClient.Get(ctx, key, deployment)  // NotFound — controller hasn't acted yet
Expect(deployment.Spec.Replicas).To(Equal(ptr(int32(2))))

// CORRECT — poll until the expected state exists
Eventually(func() error {
    return k8sClient.Get(ctx, key, deployment)
}, timeout, interval).Should(Succeed())
Expect(deployment.Spec.Replicas).To(Equal(ptr(int32(2))))

Eventually with a 10-second timeout and 100ms polling interval is standard. The controller running locally should reconcile within milliseconds, but the 10-second window handles slow CI environments.

Testing Deletion

It("should remove the finalizer and allow deletion", func() {
    // Delete the AppStack
    Expect(k8sClient.Delete(ctx, appStack)).To(Succeed())

    // After the finalizer is removed, the resource should be gone
    Eventually(func() bool {
        err := k8sClient.Get(ctx, types.NamespacedName{
            Name: appStack.Name, Namespace: namespace,
        }, &appsv1alpha1.AppStack{})
        return apierrors.IsNotFound(err)
    }, timeout, interval).Should(BeTrue())
})

Testing Self-Healing

An operator that doesn't re-create deleted resources is just declarative junk. Test it:

It("should recreate the Deployment if it is deleted externally", func() {
    // Wait for initial Deployment
    deployment := &appsv1.Deployment{}
    Eventually(func() error {
        return k8sClient.Get(ctx, types.NamespacedName{
            Name: appStack.Name, Namespace: namespace,
        }, deployment)
    }, timeout, interval).Should(Succeed())

    // Delete it manually
    Expect(k8sClient.Delete(ctx, deployment)).To(Succeed())

    // Operator should recreate it
    newDeployment := &appsv1.Deployment{}
    Eventually(func() error {
        return k8sClient.Get(ctx, types.NamespacedName{
            Name: appStack.Name, Namespace: namespace,
        }, newDeployment)
    }, timeout, interval).Should(Succeed())

    Expect(newDeployment.UID).NotTo(Equal(deployment.UID))  // new resource, new UID
})

---

Unit Testing the Build Functions

The buildDeployment, buildService, and buildHPA functions are pure functions — given an AppStack, they return a Kubernetes resource. These are ideal for table-driven unit tests:

package controller_test

import (
    "testing"

    "k8s.io/apimachinery/pkg/runtime"
    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/require"

    appsv1alpha1 "github.com/htunn/appstack-operator/api/v1alpha1"
    "github.com/htunn/appstack-operator/internal/controller"
)

func TestBuildDeployment(t *testing.T) {
    r := &controller.AppStackReconciler{
        Scheme: runtime.NewScheme(),
    }

    tests := []struct {
        name     string
        appStack *appsv1alpha1.AppStack
        wantImage    string
        wantReplicas int32
        wantPort     int32
    }{
        {
            name: "basic deployment",
            appStack: &appsv1alpha1.AppStack{
                Spec: appsv1alpha1.AppStackSpec{
                    Image:    "ghcr.io/htunn/api:v1.0.0",
                    Port:     8080,
                    Replicas: ptr(int32(3)),
                },
            },
            wantImage:    "ghcr.io/htunn/api:v1.0.0",
            wantReplicas: 3,
            wantPort:     8080,
        },
        {
            name: "default replicas when nil",
            appStack: &appsv1alpha1.AppStack{
                Spec: appsv1alpha1.AppStackSpec{
                    Image: "ghcr.io/htunn/api:v1.0.0",
                    Port:  3000,
                },
            },
            wantImage:    "ghcr.io/htunn/api:v1.0.0",
            wantReplicas: 1,
            wantPort:     3000,
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            deployment := r.BuildDeployment(tt.appStack)
            require.Len(t, deployment.Spec.Template.Spec.Containers, 1)
            assert.Equal(t, tt.wantImage, deployment.Spec.Template.Spec.Containers[0].Image)
            assert.Equal(t, tt.wantReplicas, *deployment.Spec.Replicas)
            assert.Equal(t, tt.wantPort, deployment.Spec.Template.Spec.Containers[0].Ports[0].ContainerPort)
        })
    }
}

For build functions to be testable from the _test package, export them (or test from the same package). I use the convention of exporting build functions and keeping reconcile logic unexported.

---

Running Tests in CI

The generated Makefile includes make test:

ENVTEST_K8S_VERSION = 1.29.0
ENVTEST = $(LOCALBIN)/setup-envtest

.PHONY: test
test: manifests generate fmt vet envtest
    KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" \
    go test ./... -coverprofile cover.out -timeout 5m

In GitHub Actions:

# .github/workflows/test.yml
name: Test

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Set up Go
      uses: actions/setup-go@v5
      with:
        go-version: '1.22'

    - name: Generate manifests
      run: make manifests generate

    - name: Run tests
      run: make test

    - name: Upload coverage
      uses: codecov/codecov-action@v4
      with:
        file: ./cover.out

The setup-envtest tool downloads the API server binaries on first run. They're cached between CI runs if you cache the $(LOCALBIN) directory.

---

What I Test and What I Don't

I test:

• Happy path: create CR → owned resources are created with the correct spec

• Update path: change spec → owned resources are updated

• Deletion: CR deleted → finalizer removed

• Self-healing: owned resource deleted externally → recreated

• HPA toggling: disable autoscaling after enabling → HPA is removed

• Status conditions: correct phase and condition values after creation

I don't test:

• Kubernetes API behavior (it's tested by upstream)

• Behavior of the Deployment itself (Kubernetes handles that)

• make manifests output (it's generated code)

• Log output (implementation detail, not behavior)

The rule I follow: if deleting the test would leave a behavioral path untested that could fail in production, the test should exist.

---

Next: RBAC, Deployment, and Production Hardening →