The Controller and Reconcile Loop

Table of Contents

1. Introduction

2. The Reconciler Struct

3. Setting Up Watches

4. The Reconcile Function Structure

5. Fetching the Custom Resource

6. Creating and Updating Owned Resources

7. Owner References and Garbage Collection

8. Handling Deletion with Finalizers

9. ctrl.Result and Requeue Strategies

10. Full Controller Implementation

---

Introduction

The controller is where your operator does its work. It watches for AppStack resources and reconciles the cluster toward the desired state described in the spec. Everything in this article is proven patterns from appstack-operator — not theoretical.

The complete controller file lives at internal/controller/appstack_controller.go. Let's build it from scratch.

---

The Reconciler Struct

The scaffold generates a minimal reconciler struct:

type AppStackReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

This is the core of your controller. client.Client is the controller-runtime API client — it handles GETs, CREATEs, UPDATEs, DELETEs, and status updates against the Kubernetes API. The embedded Client means you call its methods directly on the reconciler: r.Get(...), r.Create(...).

Add a Recorder for Kubernetes events:

type AppStackReconciler struct {
    client.Client
    Scheme   *runtime.Scheme
    Recorder record.EventRecorder
}

Register it in main.go:

if err = (&controller.AppStackReconciler{
    Client:   mgr.GetClient(),
    Scheme:   mgr.GetScheme(),
    Recorder: mgr.GetEventRecorderFor("appstack-controller"),
}).SetupWithManager(mgr); err != nil {
    setupLog.Error(err, "unable to create controller", "controller", "AppStack")
    os.Exit(1)
}

---

Setting Up Watches

SetupWithManager tells the manager what resources to watch and how to map watch events to reconcile requests:

func (r *AppStackReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appsv1alpha1.AppStack{}).
        Owns(&appsv1.Deployment{}).
        Owns(&corev1.Service{}).
        Owns(&autoscalingv2.HorizontalPodAutoscaler{}).
        Complete(r)
}

For(&AppStack{}): Watch AppStack resources. Any change to an AppStack triggers Reconcile() with that AppStack's namespaced name.

Owns(&Deployment{}): Watch Deployment resources owned by an AppStack. If the Deployment is modified or deleted, the owning AppStack is reconciled. Ownership is determined by owner references.

The Owns() calls are what make the operator self-healing. Without them, if someone manually deletes the Service your operator created, the operator would never know.

---

The Reconcile Function Structure

Every reconcile follows this skeleton:

func (r *AppStackReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // 1. Fetch the AppStack
    appStack := &appsv1alpha1.AppStack{}
    if err := r.Get(ctx, req.NamespacedName, appStack); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 2. Handle deletion
    if !appStack.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, appStack)
    }

    // 3. Add finalizer if not present
    if !controllerutil.ContainsFinalizer(appStack, FinalizerName) {
        controllerutil.AddFinalizer(appStack, FinalizerName)
        if err := r.Update(ctx, appStack); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{}, nil
    }

    // 4. Reconcile owned resources
    if err := r.reconcileDeployment(ctx, appStack); err != nil {
        log.Error(err, "failed to reconcile Deployment")
        return r.setDegradedStatus(ctx, appStack, err)
    }

    if err := r.reconcileService(ctx, appStack); err != nil {
        log.Error(err, "failed to reconcile Service")
        return r.setDegradedStatus(ctx, appStack, err)
    }

    if err := r.reconcileHPA(ctx, appStack); err != nil {
        log.Error(err, "failed to reconcile HPA")
        return r.setDegradedStatus(ctx, appStack, err)
    }

    // 5. Update status
    if err := r.updateStatus(ctx, appStack); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

This is the consistent pattern I use. Each step is responsible for one thing. Errors bubble up cleanly.

---

Fetching the Custom Resource

appStack := &appsv1alpha1.AppStack{}
if err := r.Get(ctx, req.NamespacedName, appStack); err != nil {
    return ctrl.Result{}, client.IgnoreNotFound(err)
}

client.IgnoreNotFound(err) returns nil when the resource no longer exists. This happens when:

• The resource was deleted and this reconcile was already queued before the deletion was processed

• A race condition during startup

Returning nil on NotFound is correct. Returning an error would cause the controller to requeue and retry indefinitely for a resource that doesn't exist.

Never assume the object is still valid after a Get returns without error. Always check .DeletionTimestamp immediately after fetching — the object may be in the process of being deleted.

---

Creating and Updating Owned Resources

The pattern for managing a child resource (Deployment, Service, HPA) is always the same:

func (r *AppStackReconciler) reconcileDeployment(ctx context.Context, appStack *appsv1alpha1.AppStack) error {
    log := log.FromContext(ctx)

    // Build the desired state
    desired := r.buildDeployment(appStack)

    // Set the owner reference so this Deployment is GC'd with the AppStack
    if err := controllerutil.SetControllerReference(appStack, desired, r.Scheme); err != nil {
        return err
    }

    // Fetch the current state
    existing := &appsv1.Deployment{}
    err := r.Get(ctx, types.NamespacedName{Name: desired.Name, Namespace: desired.Namespace}, existing)

    if apierrors.IsNotFound(err) {
        // Does not exist — create it
        log.Info("Creating Deployment", "name", desired.Name)
        r.Recorder.Event(appStack, corev1.EventTypeNormal, "DeploymentCreated",
            fmt.Sprintf("Created Deployment %s", desired.Name))
        return r.Create(ctx, desired)
    }

    if err != nil {
        return err
    }

    // Exists — check if it needs updating
    if deploymentNeedsUpdate(existing, desired) {
        existing.Spec = desired.Spec
        existing.Labels = desired.Labels
        log.Info("Updating Deployment", "name", existing.Name)
        r.Recorder.Event(appStack, corev1.EventTypeNormal, "DeploymentUpdated",
            fmt.Sprintf("Updated Deployment %s", existing.Name))
        return r.Update(ctx, existing)
    }

    return nil
}

Building the Desired Deployment

func (r *AppStackReconciler) buildDeployment(appStack *appsv1alpha1.AppStack) *appsv1.Deployment {
    labels := map[string]string{
        "app.kubernetes.io/name":       appStack.Name,
        "app.kubernetes.io/managed-by": "appstack-operator",
    }

    replicas := int32(1)
    if appStack.Spec.Replicas != nil {
        replicas = *appStack.Spec.Replicas
    }

    return &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      appStack.Name,
            Namespace: appStack.Namespace,
            Labels:    labels,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: labels,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: labels,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{
                        {
                            Name:  appStack.Name,
                            Image: appStack.Spec.Image,
                            Ports: []corev1.ContainerPort{
                                {
                                    ContainerPort: appStack.Spec.Port,
                                    Protocol:      corev1.ProtocolTCP,
                                },
                            },
                            Env: appStack.Spec.Env,
                            Resources: corev1.ResourceRequirements{
                                Requests: corev1.ResourceList{
                                    corev1.ResourceCPU:    resource.MustParse("100m"),
                                    corev1.ResourceMemory: resource.MustParse("128Mi"),
                                },
                                Limits: corev1.ResourceList{
                                    corev1.ResourceCPU:    resource.MustParse("500m"),
                                    corev1.ResourceMemory: resource.MustParse("256Mi"),
                                },
                            },
                            ReadinessProbe: &corev1.Probe{
                                ProbeHandler: corev1.ProbeHandler{
                                    HTTPGet: &corev1.HTTPGetAction{
                                        Path: "/healthz",
                                        Port: intstr.FromInt32(appStack.Spec.Port),
                                    },
                                },
                                InitialDelaySeconds: 5,
                                PeriodSeconds:       10,
                            },
                        },
                    },
                },
            },
        },
    }
}

Building the Service

func (r *AppStackReconciler) buildService(appStack *appsv1alpha1.AppStack) *corev1.Service {
    labels := map[string]string{
        "app.kubernetes.io/name":       appStack.Name,
        "app.kubernetes.io/managed-by": "appstack-operator",
    }

    return &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      appStack.Name,
            Namespace: appStack.Namespace,
            Labels:    labels,
        },
        Spec: corev1.ServiceSpec{
            Selector: labels,
            Ports: []corev1.ServicePort{
                {
                    Name:       "http",
                    Port:       80,
                    TargetPort: intstr.FromInt32(appStack.Spec.Port),
                    Protocol:   corev1.ProtocolTCP,
                },
            },
            Type: corev1.ServiceTypeClusterIP,
        },
    }
}

Building the HPA

func (r *AppStackReconciler) reconcileHPA(ctx context.Context, appStack *appsv1alpha1.AppStack) error {
    existing := &autoscalingv2.HorizontalPodAutoscaler{}
    err := r.Get(ctx, types.NamespacedName{
        Name:      appStack.Name,
        Namespace: appStack.Namespace,
    }, existing)

    // If autoscaling is disabled, delete the HPA if it exists
    if appStack.Spec.Autoscaling == nil || !appStack.Spec.Autoscaling.Enabled {
        if err == nil {
            // HPA exists but autoscaling was disabled — remove it
            return r.Delete(ctx, existing)
        }
        return client.IgnoreNotFound(err)
    }

    desired := r.buildHPA(appStack)
    if err := controllerutil.SetControllerReference(appStack, desired, r.Scheme); err != nil {
        return err
    }

    if apierrors.IsNotFound(err) {
        return r.Create(ctx, desired)
    }

    if err != nil {
        return err
    }

    existing.Spec = desired.Spec
    return r.Update(ctx, existing)
}

func (r *AppStackReconciler) buildHPA(appStack *appsv1alpha1.AppStack) *autoscalingv2.HorizontalPodAutoscaler {
    as := appStack.Spec.Autoscaling
    cpuTarget := as.CPUTargetPercent

    minReplicas := int32(1)
    if as.MinReplicas != nil {
        minReplicas = *as.MinReplicas
    }

    return &autoscalingv2.HorizontalPodAutoscaler{
        ObjectMeta: metav1.ObjectMeta{
            Name:      appStack.Name,
            Namespace: appStack.Namespace,
        },
        Spec: autoscalingv2.HorizontalPodAutoscalerSpec{
            ScaleTargetRef: autoscalingv2.CrossVersionObjectReference{
                APIVersion: "apps/v1",
                Kind:       "Deployment",
                Name:       appStack.Name,
            },
            MinReplicas: &minReplicas,
            MaxReplicas: as.MaxReplicas,
            Metrics: []autoscalingv2.MetricSpec{
                {
                    Type: autoscalingv2.ResourceMetricSourceType,
                    Resource: &autoscalingv2.ResourceMetricSource{
                        Name: corev1.ResourceCPU,
                        Target: autoscalingv2.MetricTarget{
                            Type:               autoscalingv2.UtilizationMetricType,
                            AverageUtilization: &cpuTarget,
                        },
                    },
                },
            },
        },
    }
}

---

Owner References and Garbage Collection

controllerutil.SetControllerReference(owner, owned, scheme) sets the ownerReferences field on the child resource. When the AppStack is deleted, Kubernetes automatically garbage-collects all owned resources (Deployment, Service, HPA).

# What the Deployment looks like after SetControllerReference
metadata:
  ownerReferences:
  - apiVersion: apps.htunn.io/v1alpha1
    kind: AppStack
    name: api-service
    uid: "abc-123"
    controller: true
    blockOwnerDeletion: true

controller: true means only one owner can be the controller. blockOwnerDeletion: true means the child blocks the parent from being fully deleted until the child is gone. This is appropriate for all owned resources.

Important: Owner references only work within the same namespace. Cross-namespace ownership is not supported by the Kubernetes GC. All resources created by AppStack must be in the same namespace.

---

Handling Deletion with Finalizers

Kubernetes finalizers let your controller run cleanup logic before the resource is garbage collected. Without a finalizer, the resource is deleted immediately.

For AppStack, I add a finalizer to handle cases where cleanup needs to happen in a specific order — for example, waiting for in-flight requests to drain before removing the Service.

const FinalizerName = "apps.htunn.io/appstack-finalizer"

// In Reconcile(), after the initial fetch:
if !appStack.DeletionTimestamp.IsZero() {
    return r.handleDeletion(ctx, appStack)
}

if !controllerutil.ContainsFinalizer(appStack, FinalizerName) {
    controllerutil.AddFinalizer(appStack, FinalizerName)
    return ctrl.Result{}, r.Update(ctx, appStack)
}

func (r *AppStackReconciler) handleDeletion(ctx context.Context, appStack *appsv1alpha1.AppStack) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    if controllerutil.ContainsFinalizer(appStack, FinalizerName) {
        // Run cleanup here — e.g., drain connections, deregister from external systems

        log.Info("Running deletion cleanup", "name", appStack.Name)
        r.Recorder.Event(appStack, corev1.EventTypeNormal, "Deleting",
            "AppStack is being deleted, running cleanup")

        // Remove the finalizer so Kubernetes proceeds with deletion
        controllerutil.RemoveFinalizer(appStack, FinalizerName)
        if err := r.Update(ctx, appStack); err != nil {
            return ctrl.Result{}, err
        }
    }

    // Owned resources (Deployment, Service, HPA) are cleaned up by GC via ownerReferences
    return ctrl.Result{}, nil
}

If AppStack only owns native Kubernetes resources, owner references handle cleanup automatically — you don't strictly need a finalizer. Finalizers are necessary when you have external resources to clean up (e.g., cloud load balancers, external DNS records, database users).

---

ctrl.Result and Requeue Strategies

The return value from Reconcile() controls what the controller does next:

// Success — do nothing until the next watch event
return ctrl.Result{}, nil

// Requeue immediately — controller decides the backoff
return ctrl.Result{}, fmt.Errorf("something failed")

// Requeue after a fixed duration — useful for polling external state
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

// Requeue now without an error (rare)
return ctrl.Result{Requeue: true}, nil

When to use each:

Return	Use when
{}, nil	Reconcile completed successfully
{}, err	A real error occurred — controller-runtime applies exponential backoff
{RequeueAfter: N}, nil	You need to poll (e.g., waiting for an external resource)
{Requeue: true}, nil	You made a change (e.g., added finalizer) and need to re-enter reconcile immediately without triggering a watch event

The most common mistake is returning {}, err for expected transient conditions (like a resource not being ready yet). This floods the controller with retries. Use {RequeueAfter: 10s}, nil for known wait conditions.

---

Full Controller Implementation

Here is the complete appstack_controller.go reflecting all the patterns above:

package controller

import (
    "context"
    "fmt"
    "time"

    appsv1 "k8s.io/api/apps/v1"
    autoscalingv2 "k8s.io/api/autoscaling/v2"
    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/resource"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    "k8s.io/apimachinery/pkg/util/intstr"
    "k8s.io/client-go/tools/record"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/log"

    appsv1alpha1 "github.com/htunn/appstack-operator/api/v1alpha1"
)

const FinalizerName = "apps.htunn.io/appstack-finalizer"

// +kubebuilder:rbac:groups=apps.htunn.io,resources=appstacks,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.htunn.io,resources=appstacks/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps.htunn.io,resources=appstacks/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=events,verbs=create;patch

type AppStackReconciler struct {
    client.Client
    Scheme   *runtime.Scheme
    Recorder record.EventRecorder
}

func (r *AppStackReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    appStack := &appsv1alpha1.AppStack{}
    if err := r.Get(ctx, req.NamespacedName, appStack); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    if !appStack.DeletionTimestamp.IsZero() {
        return r.handleDeletion(ctx, appStack)
    }

    if !controllerutil.ContainsFinalizer(appStack, FinalizerName) {
        controllerutil.AddFinalizer(appStack, FinalizerName)
        if err := r.Update(ctx, appStack); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{}, nil
    }

    if err := r.reconcileDeployment(ctx, appStack); err != nil {
        log.Error(err, "failed to reconcile Deployment")
        return r.setDegradedStatus(ctx, appStack, ReasonReconcileError, err.Error())
    }

    if err := r.reconcileService(ctx, appStack); err != nil {
        log.Error(err, "failed to reconcile Service")
        return r.setDegradedStatus(ctx, appStack, ReasonReconcileError, err.Error())
    }

    if err := r.reconcileHPA(ctx, appStack); err != nil {
        log.Error(err, "failed to reconcile HPA")
        return r.setDegradedStatus(ctx, appStack, ReasonReconcileError, err.Error())
    }

    return ctrl.Result{}, r.updateStatus(ctx, appStack)
}

func (r *AppStackReconciler) reconcileDeployment(ctx context.Context, appStack *appsv1alpha1.AppStack) error {
    desired := r.buildDeployment(appStack)
    if err := controllerutil.SetControllerReference(appStack, desired, r.Scheme); err != nil {
        return err
    }

    existing := &appsv1.Deployment{}
    err := r.Get(ctx, types.NamespacedName{Name: desired.Name, Namespace: desired.Namespace}, existing)

    if apierrors.IsNotFound(err) {
        r.Recorder.Event(appStack, corev1.EventTypeNormal, "DeploymentCreated",
            fmt.Sprintf("Created Deployment %s", desired.Name))
        return r.Create(ctx, desired)
    }
    if err != nil {
        return err
    }

    if existing.Spec.Template.Spec.Containers[0].Image != desired.Spec.Template.Spec.Containers[0].Image ||
        *existing.Spec.Replicas != *desired.Spec.Replicas {
        existing.Spec = desired.Spec
        r.Recorder.Event(appStack, corev1.EventTypeNormal, "DeploymentUpdated",
            fmt.Sprintf("Updated Deployment %s", existing.Name))
        return r.Update(ctx, existing)
    }

    return nil
}

func (r *AppStackReconciler) reconcileService(ctx context.Context, appStack *appsv1alpha1.AppStack) error {
    desired := r.buildService(appStack)
    if err := controllerutil.SetControllerReference(appStack, desired, r.Scheme); err != nil {
        return err
    }

    existing := &corev1.Service{}
    err := r.Get(ctx, types.NamespacedName{Name: desired.Name, Namespace: desired.Namespace}, existing)

    if apierrors.IsNotFound(err) {
        r.Recorder.Event(appStack, corev1.EventTypeNormal, "ServiceCreated",
            fmt.Sprintf("Created Service %s", desired.Name))
        return r.Create(ctx, desired)
    }

    return err
}

func (r *AppStackReconciler) updateStatus(ctx context.Context, appStack *appsv1alpha1.AppStack) error {
    deployment := &appsv1.Deployment{}
    err := r.Get(ctx, types.NamespacedName{Name: appStack.Name, Namespace: appStack.Namespace}, deployment)
    if err != nil {
        return client.IgnoreNotFound(err)
    }

    appStack.Status.ReadyReplicas = deployment.Status.ReadyReplicas
    appStack.Status.ObservedGeneration = appStack.Generation

    if deployment.Status.ReadyReplicas > 0 {
        appStack.Status.Phase = appsv1alpha1.AppStackPhaseRunning
        meta.SetStatusCondition(&appStack.Status.Conditions, metav1.Condition{
            Type:               appsv1alpha1.ConditionTypeAvailable,
            Status:             metav1.ConditionTrue,
            Reason:             ReasonDeploymentReady,
            Message:            fmt.Sprintf("%d/%d replicas ready", deployment.Status.ReadyReplicas, *deployment.Spec.Replicas),
            ObservedGeneration: appStack.Generation,
        })
    } else {
        if deployment.Status.UnavailableReplicas > 0 {
            appStack.Status.Phase = appsv1alpha1.AppStackPhaseDegraded
        } else {
            appStack.Status.Phase = appsv1alpha1.AppStackPhasePending
        }
        meta.SetStatusCondition(&appStack.Status.Conditions, metav1.Condition{
            Type:               appsv1alpha1.ConditionTypeAvailable,
            Status:             metav1.ConditionFalse,
            Reason:             ReasonDeploymentPending,
            Message:            "Waiting for replicas to become ready",
            ObservedGeneration: appStack.Generation,
        })
    }

    return r.Status().Update(ctx, appStack)
}

func (r *AppStackReconciler) setDegradedStatus(ctx context.Context, appStack *appsv1alpha1.AppStack, reason, message string) (ctrl.Result, error) {
    appStack.Status.Phase = appsv1alpha1.AppStackPhaseError
    meta.SetStatusCondition(&appStack.Status.Conditions, metav1.Condition{
        Type:               appsv1alpha1.ConditionTypeDegraded,
        Status:             metav1.ConditionTrue,
        Reason:             reason,
        Message:            message,
        ObservedGeneration: appStack.Generation,
    })
    if err := r.Status().Update(ctx, appStack); err != nil {
        return ctrl.Result{}, err
    }
    return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
}

func (r *AppStackReconciler) handleDeletion(ctx context.Context, appStack *appsv1alpha1.AppStack) (ctrl.Result, error) {
    if controllerutil.ContainsFinalizer(appStack, FinalizerName) {
        r.Recorder.Event(appStack, corev1.EventTypeNormal, "Deleting", "AppStack is being deleted")
        controllerutil.RemoveFinalizer(appStack, FinalizerName)
        if err := r.Update(ctx, appStack); err != nil {
            return ctrl.Result{}, err
        }
    }
    return ctrl.Result{}, nil
}

func (r *AppStackReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&appsv1alpha1.AppStack{}).
        Owns(&appsv1.Deployment{}).
        Owns(&corev1.Service{}).
        Owns(&autoscalingv2.HorizontalPodAutoscaler{}).
        Complete(r)
}

---

Next: Status, Events, and Observability →