RBAC, Deployment, and Production Hardening

Table of Contents

1. Introduction

2. RBAC Markers and Generated Roles

3. Deploying to the Cluster

4. The Operator Container Image

5. Leader Election for High Availability

6. Resource Limits and Security Context

7. Health Probes

8. Webhook Validation (Optional)

9. Production Checklist

---

Introduction

Getting an operator reconciling correctly in a kind cluster is the halfway point. Running it in production means: the right RBAC permissions, a minimal container image, leader election to avoid split-brain in multi-replica deployments, and proper hardening.

This article covers deploying appstack-operator to a real cluster and the changes that make it safe to run in production.

---

RBAC Markers and Generated Roles

The // +kubebuilder:rbac: markers in your controller file are how the operator declares the permissions it needs. make manifests reads them and generates config/rbac/role.yaml.

All RBAC markers in appstack_controller.go:

// +kubebuilder:rbac:groups=apps.htunn.io,resources=appstacks,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.htunn.io,resources=appstacks/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps.htunn.io,resources=appstacks/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=events,verbs=create;patch

After make manifests, config/rbac/role.yaml contains:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: manager-role
rules:
- apiGroups: ["apps.htunn.io"]
  resources: ["appstacks"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps.htunn.io"]
  resources: ["appstacks/status"]
  verbs: ["get", "update", "patch"]
- apiGroups: ["apps.htunn.io"]
  resources: ["appstacks/finalizers"]
  verbs: ["update"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "patch"]

ClusterRole vs Role: The scaffold generates a ClusterRole by default because controllers watch resources across all namespaces. If your operator is namespace-scoped (only watches resources in one namespace), you can restrict it to a Role — but most operators use ClusterRole bound with a ClusterRoleBinding.

Principle of Least Privilege

Only request the verbs you actually use:

• If the controller never deletes a resource directly (relying on owner-reference GC instead), remove delete from that resource

• Never request * (all verbs) — it makes auditing impossible

• Avoid secrets access unless absolutely necessary; prefer ConfigMap for non-sensitive config

---

Deploying to the Cluster

Build the Container Image

The generated Dockerfile is production-ready:

# Build stage
FROM golang:1.22 AS builder
ARG TARGETOS
ARG TARGETARCH

WORKDIR /workspace
COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download

COPY cmd/main.go cmd/main.go
COPY api/ api/
COPY internal/ internal/

RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager cmd/main.go

# Final image — distroless: no shell, no package manager
FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532

ENTRYPOINT ["/manager"]

The distroless base image is important:

• No shell means no shell injection attacks

• USER 65532:65532 (nonroot) — the controller runs as a non-root user

Build and push:

export IMG=ghcr.io/htunn/appstack-operator:v0.1.0

make docker-build IMG=${IMG}
make docker-push IMG=${IMG}

For multi-arch builds (ARM64 + AMD64):

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag ${IMG} \
  --push .

Deploy with make deploy

make deploy IMG=${IMG}

This runs kustomize build config/default | kubectl apply -f -, which applies:

• CRDs

• Namespace (appstack-system)

• ServiceAccount

• ClusterRole + ClusterRoleBinding

• Manager Deployment

Verify:

kubectl get all -n appstack-system

NAME                                               READY   STATUS    RESTARTS
pod/appstack-controller-manager-7d8b4b9f5-xq2rn   1/1     Running   0

NAME                                                  READY   UP-TO-DATE
deployment.apps/appstack-controller-manager           1/1     1

Check the Controller Logs

kubectl logs -n appstack-system deployment/appstack-controller-manager -f

Apply a test CR:

kubectl apply -f config/samples/apps_v1alpha1_appstack.yaml
kubectl get appstack -A
kubectl describe appstack api-service -n default

---

The Operator Container Image

Version Tagging

Don't use latest for operator images in production. Use immutable semver tags:

ghcr.io/htunn/appstack-operator:v0.1.0

The Deployment in config/manager/manager.yaml references the image:

containers:
- name: manager
  image: controller:latest   # replaced by `make deploy IMG=...`

kubebuilder sets imagePullPolicy: Always by default for latest. For versioned tags, set imagePullPolicy: IfNotPresent to avoid unnecessary pulls.

Signing Images

For production, sign your images with cosign:

cosign sign ghcr.io/htunn/appstack-operator:v0.1.0

---

Leader Election for High Availability

Running a single operator replica is a single point of failure. A crashed pod means no reconciliation until it restarts. Running multiple replicas without coordination causes split-brain: two controllers reconciling the same resource simultaneously and overwriting each other's writes.

Leader election solves this. Only the leader pod actively reconciles. Follower pods watch the lease but don't act. If the leader dies, a follower acquires the lease within seconds.

Enable it in cmd/main.go:

mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
    Scheme:                 scheme,
    LeaderElection:         true,
    LeaderElectionID:       "appstack-operator-leader.htunn.io",
    LeaderElectionNamespace: "appstack-system",
    // ...
})

And pass --leader-elect=true to the manager binary (set in the Deployment args):

# config/manager/manager.yaml
args:
- --leader-elect

The lease object is stored in a Lease resource in the operator namespace:

kubectl get lease -n appstack-system
NAME                                   HOLDER                               AGE
appstack-operator-leader.htunn.io      appstack-controller-manager-7d8b4b9f5-xq2rn   2d

With leader election, you can run 2+ replicas:

# config/manager/manager.yaml
replicas: 2

Replicas beyond 2 don't add redundancy value (the lease still only has one holder). 2 replicas provides failover.

RBAC for leader election: The controller needs permission to manage Lease objects. The scaffold includes this:

// +kubebuilder:rbac:groups=coordination.k8s.io,resources=leases,verbs=get;list;watch;create;update;patch;delete

---

Resource Limits and Security Context

The generated Deployment has placeholder resource limits. Set them based on actual usage observed during development:

# config/manager/manager.yaml
resources:
  limits:
    cpu: 500m
    memory: 128Mi
  requests:
    cpu: 10m
    memory: 64Mi

For a controller with a small number of watched objects (hundreds, not thousands), 500m CPU and 128Mi memory is generous. Controller-runtime has an efficient cache — memory use is proportional to the number of cached objects.

Security Context

The Dockerfile already runs as nonroot. Mirror this in the pod spec:

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

Set this in config/manager/manager.yaml. These settings align with the Pod Security Standards restricted policy, which is enforced in most hardened clusters.

---

Health Probes

The manager exposes health endpoints at :8081:

• GET /healthz — liveness probe (returns 200 if the manager goroutine is alive)

• GET /readyz — readiness probe (returns 200 when the cache is synced and the manager is ready to reconcile)

These are already registered in main.go by the scaffold:

mgr.AddHealthzCheck("healthz", healthz.Ping)
mgr.AddReadyzCheck("readyz", healthz.Ping)

The Deployment configures the probes:

# config/manager/manager.yaml
livenessProbe:
  httpGet:
    path: /healthz
    port: 8081
  initialDelaySeconds: 15
  periodSeconds: 20
readinessProbe:
  httpGet:
    path: /readyz
    port: 8081
  initialDelaySeconds: 5
  periodSeconds: 10

The readiness probe is critical. When the pod starts, the controller-runtime cache needs to sync all watched resources before the controller starts reconciling. The readiness probe ensures traffic doesn't route to the pod until the cache is warm.

---

Webhook Validation (Optional)

For stricter validation than kubebuilder markers allow, implement a validating webhook. This lets you write Go code that runs when a CR is applied and rejects invalid resources before they reach the controller.

Scaffold a webhook:

kubebuilder create webhook \
  --group apps \
  --version v1alpha1 \
  --kind AppStack \
  --defaulting \
  --programmatic-validation

This generates api/v1alpha1/appstack_webhook.go. Implement the ValidateCreate, ValidateUpdate, and ValidateDelete methods:

func (r *AppStack) ValidateCreate() (admission.Warnings, error) {
    // Custom validation: if autoscaling is enabled, maxReplicas must be >= minReplicas
    if r.Spec.Autoscaling != nil && r.Spec.Autoscaling.Enabled {
        if r.Spec.Autoscaling.MinReplicas != nil &&
            r.Spec.Autoscaling.MaxReplicas < *r.Spec.Autoscaling.MinReplicas {
            return nil, fmt.Errorf(
                "autoscaling.maxReplicas (%d) must be >= minReplicas (%d)",
                r.Spec.Autoscaling.MaxReplicas,
                *r.Spec.Autoscaling.MinReplicas,
            )
        }
    }
    return nil, nil
}

Webhooks require TLS certificates. In production, use cert-manager:

# Install cert-manager (one-time cluster setup)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

Uncomment the cert-manager integration in config/default/kustomization.yaml. This wires up certificate generation and injection automatically.

---

Production Checklist

Before running appstack-operator in a production cluster:

RBAC

• Only requested verbs are those actually used

• No * wildcards

• No secrets access unless necessary

Container

• Distroless base image

• Runs as non-root (USER 65532:65532)

• Immutable image tag (semver, not latest)

• Image signed with cosign

Deployment

• Resource limits set

• readOnlyRootFilesystem: true

• allowPrivilegeEscalation: false

• Leader election enabled with 2 replicas

• Liveness and readiness probes configured

Observability

• Status conditions populated on every reconcile path

• Kubernetes events recorded for Create/Update/Delete/Error actions

• Metrics endpoint reachable by Prometheus

• Structured logging with logr

Operations

• make test passes in CI

• CRD versioning strategy defined (upgrade path for v1alpha1 → v1beta1)

• Controller gracefully handles not-found, conflict, and rate-limited errors

---

Previous: Testing with envtest ← Series start: README ←