Namespaces and RBAC

Introduction

I learned about Kubernetes namespaces the hard way when a junior developer accidentally deleted a production deployment—because we were running everything in the default namespace with cluster-admin privileges. That incident forced me to completely redesign our cluster security model, implementing proper namespace isolation and role-based access control (RBAC).

Since then, I've architected multi-tenant Kubernetes platforms for organizations with hundreds of teams, each requiring isolated environments with specific access controls. I've debugged mysterious permission issues at 2 AM and designed sophisticated RBAC policies for compliance requirements. Through these experiences, I've developed a deep understanding of how to properly secure and organize Kubernetes clusters.

In this comprehensive guide, I'll share everything I've learned about namespaces, RBAC, service accounts, and multi-tenancy patterns in Kubernetes.

Table of Contents

1. Understanding Namespaces

2. Namespace Basics

3. Introduction to RBAC

4. Roles and ClusterRoles

5. RoleBindings and ClusterRoleBindings

6. Service Accounts

7. Multi-Tenancy Patterns

8. Resource Quotas and Limit Ranges

9. Network Policies for Isolation

10. Security Best Practices

11. Troubleshooting Access Issues

12. Advanced RBAC Patterns

13. What I Learned

Understanding Namespaces

Namespaces provide a mechanism for isolating groups of resources within a single cluster. They're the foundation for multi-tenancy, allowing multiple teams or projects to share a cluster while maintaining separation. Think of namespaces as virtual clusters within a physical cluster.

Why Namespaces Matter

Without namespaces, every resource name must be unique across the entire cluster. As clusters grow, this becomes unmanageable. Namespaces solve this by providing scope for names, allowing different teams to use the same resource names in their respective namespaces.

Default Namespaces

Kubernetes clusters start with several default namespaces:

• default: For resources without a specified namespace

• kube-system: For Kubernetes system components

• kube-public: Readable by all users, used for public resources

• kube-node-lease: For node heartbeat objects (improved node controller performance)

Namespace Basics

Creating Namespaces

Using kubectl:

# Create namespace
kubectl create namespace development

# Create from YAML
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: staging
  labels:
    environment: staging
    team: platform
EOF

# List namespaces
kubectl get namespaces

# Describe namespace
kubectl describe namespace development

Namespace YAML with annotations:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: production
    team: platform
    cost-center: "engineering"
  annotations:
    contact: "[email protected]"
    description: "Production environment for customer-facing services"

Working with Namespaces

# Set default namespace for kubectl context
kubectl config set-context --current --namespace=production

# View current namespace
kubectl config view --minify | grep namespace:

# Run command in specific namespace
kubectl get pods -n development

# Get resources from all namespaces
kubectl get pods --all-namespaces
# Or shorthand
kubectl get pods -A

# Delete namespace (deletes all resources in it!)
kubectl delete namespace development

Deploying Resources to Namespaces

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: production  # Explicitly specify namespace
  labels:
    app: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: webapp
        image: webapp:1.0
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: webapp
  namespace: production
spec:
  selector:
    app: webapp
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP

Deploying to namespaces:

# Apply to specific namespace
kubectl apply -f webapp.yaml -n production

# If namespace in YAML, that takes precedence
kubectl apply -f webapp.yaml  # Uses namespace from YAML

Introduction to RBAC

Role-Based Access Control (RBAC) regulates access to Kubernetes resources based on roles assigned to users, groups, or service accounts.

RBAC Architecture

RBAC Components

1. Role/ClusterRole: Defines permissions (what actions on what resources)

2. RoleBinding/ClusterRoleBinding: Grants permissions to subjects

3. Subjects: Users, groups, or service accounts

Checking RBAC Permissions

# Check if you can perform an action
kubectl auth can-i create deployments
kubectl auth can-i delete pods -n production

# Check permissions for another user
kubectl auth can-i get pods [email protected]
kubectl auth can-i delete deployments [email protected] -n production

# List all permissions for current user
kubectl auth can-i --list

# List permissions in specific namespace
kubectl auth can-i --list -n production

Roles and ClusterRoles

Role (Namespace-scoped)

A Role grants permissions within a single namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: development
  name: pod-reader
rules:
- apiGroups: [""]  # "" indicates core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]

Common verbs:

• get: Read individual resource

• list: List multiple resources

• watch: Watch for changes

• create: Create new resources

• update: Update existing resources

• patch: Partially update resources

• delete: Delete resources

• deletecollection: Delete multiple resources

Developer role example:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: development
  name: developer
rules:
# Pods
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
# Deployments
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
# Services
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
# ConfigMaps and Secrets
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]  # Read-only for secrets

ClusterRole (Cluster-wide)

ClusterRoles can grant permissions to:

• Cluster-scoped resources (nodes, persistent volumes)

• Non-resource endpoints (/healthz, /metrics)

• Resources across all namespaces

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-monitoring
rules:
# Read nodes
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
# Read persistent volumes
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["get", "list", "watch"]
# Access metrics endpoints
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
# Read pods across all namespaces
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

View built-in ClusterRoles:

# List all ClusterRoles
kubectl get clusterroles

# View specific ClusterRole
kubectl describe clusterrole view
kubectl describe clusterrole edit
kubectl describe clusterrole admin
kubectl describe clusterrole cluster-admin

Aggregated ClusterRoles

ClusterRoles can aggregate permissions from other roles:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-aggregate
aggregationRule:
  clusterRoleSelectors:
  - matchLabels:
      rbac.example.com/aggregate-to-monitoring: "true"
rules: []  # Rules auto-populated from aggregated roles
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-pods
  labels:
    rbac.example.com/aggregate-to-monitoring: "true"
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-metrics
  labels:
    rbac.example.com/aggregate-to-monitoring: "true"
rules:
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

RoleBindings and ClusterRoleBindings

RoleBinding

Grants a Role's permissions to subjects within a namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-binding
  namespace: development
subjects:
# User
- kind: User
  name: [email protected]
  apiGroup: rbac.authorization.k8s.io
# Group
- kind: Group
  name: developers
  apiGroup: rbac.authorization.k8s.io
# Service Account
- kind: ServiceAccount
  name: ci-runner
  namespace: development
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io

Creating RoleBinding with kubectl:

# Bind role to user
kubectl create rolebinding developer-jane \
  --role=developer \
  [email protected] \
  -n development

# Bind role to service account
kubectl create rolebinding ci-runner \
  --role=developer \
  --serviceaccount=development:ci-runner \
  -n development

# Bind ClusterRole in namespace scope
kubectl create rolebinding pod-reader \
  --clusterrole=view \
  [email protected] \
  -n production

ClusterRoleBinding

Grants a ClusterRole's permissions across all namespaces:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-admin-binding
subjects:
- kind: User
  name: [email protected]
  apiGroup: rbac.authorization.k8s.io
- kind: Group
  name: platform-team
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Important: ClusterRoleBindings grant permissions cluster-wide. Use carefully!

# Create cluster-wide binding
kubectl create clusterrolebinding cluster-monitoring \
  --clusterrole=cluster-monitoring \
  [email protected]

# List all bindings
kubectl get rolebindings -A
kubectl get clusterrolebindings

Service Accounts

Service accounts provide identities for pods to interact with the Kubernetes API.

Creating Service Accounts

apiVersion: v1
kind: ServiceAccount
metadata:
  name: webapp-sa
  namespace: production
  annotations:
    description: "Service account for webapp pods"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: webapp-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["webapp-secrets"]  # Specific secret only
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: webapp-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: webapp-sa
  namespace: production
roleRef:
  kind: Role
  name: webapp-role
  apiGroup: rbac.authorization.k8s.io

Using Service Accounts in Pods

apiVersion: v1
kind: Pod
metadata:
  name: webapp
  namespace: production
spec:
  serviceAccountName: webapp-sa
  automountServiceAccountToken: true  # Default: true
  containers:
  - name: webapp
    image: webapp:1.0
    # Token automatically mounted at /var/run/secrets/kubernetes.io/serviceaccount/token

Accessing Kubernetes API from pod:

# Exec into pod
kubectl exec -it webapp -n production -- /bin/sh

# Token location
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
CA_CERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)

# Call API
curl --cacert $CA_CERT \
  -H "Authorization: Bearer $TOKEN" \
  https://kubernetes.default.svc/api/v1/namespaces/$NAMESPACE/configmaps

Disable Automounting Token

For pods that don't need API access:

apiVersion: v1
kind: Pod
metadata:
  name: static-webapp
spec:
  serviceAccountName: default
  automountServiceAccountToken: false  # Disable for security
  containers:
  - name: webapp
    image: nginx:1.21

Multi-Tenancy Patterns

Namespace-per-Team

# Team A namespace
apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    team: team-a
---
# Team B namespace
apiVersion: v1
kind: Namespace
metadata:
  name: team-b
  labels:
    team: team-b
---
# Team A admin role
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-a-admin
  namespace: team-a
subjects:
- kind: Group
  name: team-a-members
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: admin  # Built-in admin role
  apiGroup: rbac.authorization.k8s.io

Namespace-per-Environment

# Development
apiVersion: v1
kind: Namespace
metadata:
  name: webapp-dev
  labels:
    app: webapp
    environment: development
---
# Staging
apiVersion: v1
kind: Namespace
metadata:
  name: webapp-staging
  labels:
    app: webapp
    environment: staging
---
# Production
apiVersion: v1
kind: Namespace
metadata:
  name: webapp-prod
  labels:
    app: webapp
    environment: production

Cross-Namespace Access

# Allow service in namespace-a to access service in namespace-b
apiVersion: v1
kind: Service
metadata:
  name: database
  namespace: database-ns
spec:
  selector:
    app: postgres
  ports:
  - port: 5432
---
# Access from different namespace using FQDN
# database.database-ns.svc.cluster.local

Resource Quotas and Limit Ranges

Resource Quotas

Limit aggregate resource consumption per namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: development
spec:
  hard:
    # Compute resources
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    
    # Storage resources
    requests.storage: 100Gi
    persistentvolumeclaims: "10"
    
    # Object counts
    pods: "50"
    services: "10"
    secrets: "20"
    configmaps: "20"
    replicationcontrollers: "10"
    services.loadbalancers: "2"
    services.nodeports: "5"

Checking quota usage:

# View quota
kubectl describe resourcequota compute-quota -n development

# Output shows used vs hard limits
# Name:                   compute-quota
# Namespace:              development
# Resource                Used   Hard
# --------                ----   ----
# limits.cpu              8      20
# limits.memory           16Gi   40Gi
# pods                    25     50
# requests.cpu            4      10
# requests.memory         8Gi    20Gi

Limit Ranges

Set default and maximum resource limits for pods:

apiVersion: v1
kind: LimitRange
metadata:
  name: resource-limits
  namespace: development
spec:
  limits:
  # Pod limits
  - type: Pod
    max:
      cpu: "4"
      memory: 8Gi
    min:
      cpu: "100m"
      memory: 128Mi
  
  # Container limits
  - type: Container
    default:  # Default limits if not specified
      cpu: "500m"
      memory: 512Mi
    defaultRequest:  # Default requests if not specified
      cpu: "100m"
      memory: 128Mi
    max:
      cpu: "2"
      memory: 4Gi
    min:
      cpu: "50m"
      memory: 64Mi
  
  # PVC limits
  - type: PersistentVolumeClaim
    max:
      storage: 50Gi
    min:
      storage: 1Gi

Effect on pods without resource specifications:

# Pod without resources defined
apiVersion: v1
kind: Pod
metadata:
  name: auto-limited
  namespace: development
spec:
  containers:
  - name: app
    image: myapp:1.0
    # LimitRange automatically adds:
    # resources:
    #   requests:
    #     cpu: 100m
    #     memory: 128Mi
    #   limits:
    #     cpu: 500m
    #     memory: 512Mi

Network Policies for Isolation

Network Policies provide network-level isolation between namespaces:

# Deny all ingress traffic by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}  # Apply to all pods
  policyTypes:
  - Ingress
---
# Allow specific ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-webapp-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: webapp
  policyTypes:
  - Ingress
  ingress:
  # Allow from ingress controller
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 8080
  # Allow from same namespace
  - from:
    - podSelector: {}
    ports:
    - protocol: TCP
      port: 8080

Cross-namespace network policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-frontend
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
      podSelector:
        matchLabels:
          app: webapp
    ports:
    - protocol: TCP
      port: 8080

Security Best Practices

1. Principle of Least Privilege

# BAD: Granting cluster-admin
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: developer-binding
subjects:
- kind: Group
  name: developers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin  # Too permissive!
  apiGroup: rbac.authorization.k8s.io

# GOOD: Specific permissions only
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-binding
  namespace: development
subjects:
- kind: Group
  name: developers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer-role
  apiGroup: rbac.authorization.k8s.io

2. Use Service Accounts for Pods

# Create dedicated service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
---
# Grant minimum required permissions
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]

3. Namespace Isolation

# Enforce namespace for resources
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: development
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
# Binding limited to namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: developer-binding
  namespace: development  # Only affects development namespace
subjects:
- kind: Group
  name: developers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io

4. Audit RBAC Regularly

# List all RoleBindings and ClusterRoleBindings
kubectl get rolebindings -A -o wide
kubectl get clusterrolebindings -o wide

# Find users with cluster-admin
kubectl get clusterrolebindings -o json | \
  jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .metadata.name'

# Check who can perform dangerous actions
kubectl get clusterrolebindings -o json | \
  jq -r '.items[] | select(.roleRef.name | contains("admin"))'

5. Use ResourceNames for Fine-Grained Control

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["app-secrets", "database-credentials"]  # Specific secrets only
  verbs: ["get"]

Troubleshooting Access Issues

Permission Denied Errors

# Error: User cannot list pods
kubectl get pods -n production
# Error from server (Forbidden): pods is forbidden: User "[email protected]" cannot list resource "pods"

# Debug: Check permissions
kubectl auth can-i list pods -n production [email protected]
# no

# Check role bindings
kubectl get rolebindings -n production
kubectl describe rolebinding developer-binding -n production

# Fix: Create appropriate binding
kubectl create rolebinding jane-pod-reader \
  --role=pod-reader \
  [email protected] \
  -n production

Service Account Issues

# Pod cannot access API
kubectl logs my-pod
# Error: secrets is forbidden: User "system:serviceaccount:default:default" cannot list secrets

# Check service account
kubectl get pod my-pod -o jsonpath='{.spec.serviceAccountName}'

# Check service account permissions
kubectl auth can-i list secrets \
  --as=system:serviceaccount:default:default \
  -n default

# Create proper role and binding
kubectl create role secret-reader \
  --verb=get,list \
  --resource=secrets \
  -n default

kubectl create rolebinding default-secret-reader \
  --role=secret-reader \
  --serviceaccount=default:default \
  -n default

Debugging RBAC with impersonation

# Test as another user
kubectl get pods [email protected]
kubectl get pods [email protected] -n production

# Test as service account
kubectl get secrets \
  --as=system:serviceaccount:production:webapp-sa \
  -n production

# Test as group
kubectl get deployments --as=user --as-group=developers

Advanced RBAC Patterns

Dynamic Admission Control

Use ValidatingWebhookConfiguration for additional security:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: namespace-validator
webhooks:
- name: validate.namespace.example.com
  clientConfig:
    service:
      name: namespace-validator
      namespace: kube-system
      path: /validate
    caBundle: <base64-encoded-ca-cert>
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["namespaces"]
  admissionReviewVersions: ["v1"]
  sideEffects: None

Attribute-Based Access Control (ABAC) Alternative

While RBAC is standard, ABAC offers more flexibility:

{
  "apiVersion": "abac.authorization.kubernetes.io/v1beta1",
  "kind": "Policy",
  "spec": {
    "user": "alice",
    "namespace": "production",
    "resource": "pods",
    "readonly": true
  }
}

Hierarchical Namespaces

Using Hierarchical Namespace Controller (HNC):

apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchyConfiguration
metadata:
  name: hierarchy
  namespace: parent-namespace
spec:
  parent: ""
---
apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchyConfiguration
metadata:
  name: hierarchy
  namespace: child-namespace
spec:
  parent: parent-namespace

What I Learned

After years of implementing and managing RBAC in Kubernetes, these are my key lessons:

1. Start Restrictive: Begin with minimal permissions and add more as needed. It's easier to grant additional permissions than to revoke excessive ones after they're in use.

2. Document Everything: RBAC configurations can become complex. Document why specific permissions exist and who requested them. Future you will be grateful.

3. Use Groups: Manage permissions for groups rather than individual users. This scales better as teams grow and change.

4. Regular Audits: Schedule quarterly RBAC audits. People change roles, applications get deprecated, and permissions accumulate. Clean up regularly.

5. Namespace Strategy Matters: Choose your namespace strategy early (per-team, per-environment, per-application) and stick with it. Changing later is painful.

6. Test Permission Changes: Always test RBAC changes in non-production first. A misconfigured RBAC rule can break production deployments.

7. Beware cluster-admin: That role is powerful and tempting but should be rarely used. Create specific ClusterRoles instead.

8. Service Accounts Are Not Optional: Every pod should have a dedicated service account with minimum required permissions. The default service account is a security risk.

9. Network Policies Complete the Picture: RBAC controls API access, but network policies control network traffic. You need both for proper multi-tenancy.

10. Automation Is Key: Use GitOps and automated tools to manage RBAC. Manual kubectl commands lead to configuration drift and security gaps.

Security in Kubernetes is not just about preventing unauthorized access—it's about enabling teams to work safely and efficiently. Well-designed RBAC and namespace strategies provide both security and productivity.

The investment in properly designing your cluster's security model pays dividends every day through reduced incidents, faster debugging, and the confidence that your applications and data are properly protected.