AWS IAM in Kubernetes (EKS)

Introduction

The first time I ran a workload on Amazon EKS that needed to write to S3, I did what every tutorial told me to do back then: create an IAM user, generate an access key, store AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in a Kubernetes Secret, and inject them as environment variables. It worked. It was also a ticking time bomb.

That approach tied long-lived credentials to a secret that could be read by anyone with kubectl get secret access in that namespace. Rotation was manual. When an engineer left the team, we weren't sure which access keys to revoke. When we discovered one key had leaked into a public repository, the post-mortem was painful.

The right approach is to use IAM Roles for Service Accounts (IRSA) or the newer EKS Pod Identity — both let Kubernetes Service Accounts assume IAM roles without any long-lived credentials. This article covers both mechanisms, when to use each, and how to operate them at scale.

Table of Contents

1. The Problem with Static AWS Credentials

2. IAM Roles for Service Accounts (IRSA)

3. IRSA Setup and Configuration

4. EKS Pod Identity (The Modern Approach)

5. EKS Pod Identity Setup

6. IRSA vs EKS Pod Identity Comparison

7. Application Code Integration

8. IAM Policy Best Practices

9. Cross-Account Access

10. Multi-Cluster and Multi-Environment Patterns

11. Security Best Practices

12. Troubleshooting

13. What I Learned

---

The Problem with Static AWS Credentials

Why Static Credentials Are Dangerous

Risk	Impact
Long-lived keys (never expire)	Single compromised key = persistent breach
Manual rotation	Teams skip rotations; stale keys accumulate
Overly broad permissions	"Just give it full S3 access" anti-pattern
Shared keys across services	One breach exposes all services
No attribution	CloudTrail shows "IAM user" not "service"

---

IAM Roles for Service Accounts (IRSA)

IRSA was introduced in EKS in 2019. It works by federating the EKS cluster's OIDC provider with AWS IAM, allowing Kubernetes Service Accounts to assume IAM roles using projected service account tokens — the same mechanism as AKS Workload Identity.

How It Works

1. EKS cluster has an OIDC provider with a unique issuer URL

2. Kubernetes injects a short-lived OIDC-compatible token into pods via projected volume

3. The AWS SDK calls AssumeRoleWithWebIdentity using this token

4. AWS STS validates the token against the cluster's OIDC provider

5. STS returns temporary credentials scoped to the IAM role

6. The SDK uses these credentials (auto-rotating before expiry)

---

IRSA Setup and Configuration

Prerequisites

# Install required tools
brew install eksctl awscli

# Verify versions
eksctl version
aws --version

Step 1: Create EKS Cluster with OIDC Enabled

CLUSTER_NAME="eks-irsa-demo"
REGION="us-east-1"
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# Create cluster (OIDC is enabled by default in recent eksctl versions)
eksctl create cluster \
  --name $CLUSTER_NAME \
  --region $REGION \
  --node-type t3.medium \
  --nodes 2 \
  --nodes-min 1 \
  --nodes-max 3 \
  --with-oidc

# Update kubeconfig
aws eks update-kubeconfig --name $CLUSTER_NAME --region $REGION

# Get the OIDC issuer URL
OIDC_PROVIDER=$(aws eks describe-cluster \
  --name $CLUSTER_NAME \
  --region $REGION \
  --query "cluster.identity.oidc.issuer" \
  --output text | sed -e "s/^https:\/\///")

echo "OIDC Provider: $OIDC_PROVIDER"

Enable OIDC on an Existing Cluster

# Check if OIDC provider already exists
aws iam list-open-id-connect-providers | grep $CLUSTER_OIDC_ISSUER

# If not, create it
eksctl utils associate-iam-oidc-provider \
  --cluster $CLUSTER_NAME \
  --region $REGION \
  --approve

Step 2: Create an IAM Policy

# Create a policy for your application's needs
# Example: read/write access to a specific S3 bucket
cat > /tmp/s3-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-app-bucket",
        "arn:aws:s3:::my-app-bucket/*"
      ]
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name "s3-my-app-policy" \
  --policy-document file:///tmp/s3-policy.json

Step 3: Create IAM Role with Trust Policy

The trust policy defines which Kubernetes Service Accounts can assume this role.

NAMESPACE="my-app"
KSA_NAME="my-app-sa"

# Create trust policy
cat > /tmp/trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${NAMESPACE}:${KSA_NAME}"
        }
      }
    }
  ]
}
EOF

# Create the IAM role
aws iam create-role \
  --role-name "role-my-app-irsa" \
  --assume-role-policy-document file:///tmp/trust-policy.json

# Attach the policy
aws iam attach-role-policy \
  --role-name "role-my-app-irsa" \
  --policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/s3-my-app-policy"

# Get the role ARN
ROLE_ARN=$(aws iam get-role \
  --role-name "role-my-app-irsa" \
  --query "Role.Arn" \
  --output text)

echo "Role ARN: $ROLE_ARN"

Step 4: Create Annotated Kubernetes Service Account

kubectl create namespace $NAMESPACE

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${KSA_NAME}
  namespace: ${NAMESPACE}
  annotations:
    # This annotation links the KSA to the IAM role
    eks.amazonaws.com/role-arn: "${ROLE_ARN}"
    # Optional: set token expiry (default: 86400s = 24h)
    eks.amazonaws.com/token-expiration: "86400"
EOF

Or using eksctl for simple cases:

eksctl create iamserviceaccount \
  --cluster $CLUSTER_NAME \
  --region $REGION \
  --namespace $NAMESPACE \
  --name $KSA_NAME \
  --attach-policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/s3-my-app-policy" \
  --approve

Step 5: Deploy Your Application

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      # Reference the annotated service account
      serviceAccountName: my-app-sa
      containers:
        - name: app
          image: my-ecr-repo.dkr.ecr.us-east-1.amazonaws.com/my-app:latest
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
          # IRSA automatically injects these via the EKS Pod Identity Webhook:
          # env:
          #   - name: AWS_ROLE_ARN
          #     value: "arn:aws:iam::123456789012:role/role-my-app-irsa"
          #   - name: AWS_WEB_IDENTITY_TOKEN_FILE
          #     value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
          #   - name: AWS_DEFAULT_REGION
          #     value: "us-east-1"

After deployment, verify the webhook-injected environment variables:

kubectl exec -n my-app deployment/my-app -- env | grep AWS
# AWS_ROLE_ARN=arn:aws:iam::123456789012:role/role-my-app-irsa
# AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
# AWS_DEFAULT_REGION=us-east-1

---

EKS Pod Identity (The Modern Approach)

Launched in late 2023, EKS Pod Identity is AWS's simplified alternative to IRSA. It removes the need to manage OIDC provider configurations per-cluster and simplifies the IAM trust policy.

Key Differences from IRSA

Aspect	IRSA	EKS Pod Identity
OIDC provider required	Yes, per cluster	No
Trust policy format	References OIDC + subject	References pods.eks.amazonaws.com
Association mechanism	KSA annotation	EKS API resource
Cross-account	Supported	Supported (via role chaining)
Node agent required	No	Yes (EKS Pod Identity Agent add-on)
Portability	Any EKS cluster with OIDC	EKS only

---

EKS Pod Identity Setup

Step 1: Enable EKS Pod Identity Agent Add-on

# Install the EKS Pod Identity Agent DaemonSet
aws eks create-addon \
  --cluster-name $CLUSTER_NAME \
  --addon-name eks-pod-identity-agent \
  --region $REGION

# Wait for it to be active
aws eks wait addon-active \
  --cluster-name $CLUSTER_NAME \
  --addon-name eks-pod-identity-agent \
  --region $REGION

# Verify DaemonSet is running
kubectl get daemonset -n kube-system | grep eks-pod-identity-agent

Step 2: Create an IAM Role with Pod Identity Trust Policy

The trust policy for Pod Identity is simpler — it trusts the EKS service principal, not a cluster-specific OIDC provider.

cat > /tmp/pod-identity-trust.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}
EOF

# Create the role
aws iam create-role \
  --role-name "role-my-app-pod-identity" \
  --assume-role-policy-document file:///tmp/pod-identity-trust.json

# Attach the policy (same as IRSA example above)
aws iam attach-role-policy \
  --role-name "role-my-app-pod-identity" \
  --policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/s3-my-app-policy"

ROLE_ARN_PI=$(aws iam get-role \
  --role-name "role-my-app-pod-identity" \
  --query "Role.Arn" \
  --output text)

Step 3: Create Pod Identity Association

Instead of annotating the Service Account, you create an EKS API resource that links the KSA to the role.

# Create the association
aws eks create-pod-identity-association \
  --cluster-name $CLUSTER_NAME \
  --region $REGION \
  --namespace $NAMESPACE \
  --service-account $KSA_NAME \
  --role-arn $ROLE_ARN_PI

# List associations
aws eks list-pod-identity-associations \
  --cluster-name $CLUSTER_NAME \
  --region $REGION \
  --namespace $NAMESPACE

Step 4: Create Kubernetes Service Account (No Annotation Needed)

# service-account.yaml
# With Pod Identity, no annotation is needed on the Service Account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-sa
  namespace: my-app
  # No eks.amazonaws.com/role-arn annotation required

Step 5: Deploy Your Application

# deployment.yaml — same as IRSA, no changes needed in the pod spec
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      serviceAccountName: my-app-sa
      containers:
        - name: app
          image: my-ecr-repo.dkr.ecr.us-east-1.amazonaws.com/my-app:latest
          ports:
            - containerPort: 8080

The Pod Identity Agent DaemonSet intercepts credential requests, handles the STS exchange, and delivers credentials to pod containers automatically.

---

IRSA vs EKS Pod Identity Comparison

Detailed Comparison

Feature	IRSA	EKS Pod Identity
Maturity	GA since 2019, widely adopted	GA since 2024
OIDC provider	Required, per-cluster	Not required
Trust policy	References <oidc_provider>:sub	References pods.eks.amazonaws.com
Service account annotation	Required	Not required
Association management	Implicit (annotation)	Explicit (EKS API)
Node agent	None	Pod Identity Agent DaemonSet
Maximum sessions per role	No limit	500 per namespace/SA pair per role
Terraform support	Mature	Available via aws_eks_pod_identity_association
Cluster migration	Trust policy must be updated per cluster	No changes needed
EKS Anywhere / non-EKS	Supported	EKS only
Graviton2/ARM support	Yes	Yes

---

Application Code Integration

Go

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
)

func main() {
    ctx := context.Background()

    // LoadDefaultConfig automatically picks up:
    // - IRSA: AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE env vars
    // - Pod Identity: handled by credential provider chain
    // - Local dev: ~/.aws/credentials or environment variables
    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("us-east-1"))
    if err != nil {
        log.Fatalf("unable to load SDK config: %v", err)
    }

    // S3 example
    s3Client := s3.NewFromConfig(cfg)
    
    result, err := s3Client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: aws.String("my-app-bucket"),
        Key:    aws.String("config/settings.json"),
    })
    if err != nil {
        log.Fatalf("failed to get object: %v", err)
    }
    defer result.Body.Close()
    fmt.Println("Successfully retrieved S3 object")

    // Secrets Manager example
    smClient := secretsmanager.NewFromConfig(cfg)
    secret, err := smClient.GetSecretValue(ctx, &secretsmanager.GetSecretValueInput{
        SecretId: aws.String("my-app/database-password"),
    })
    if err != nil {
        log.Fatalf("failed to get secret: %v", err)
    }
    fmt.Printf("Secret retrieved: %s\n", aws.ToString(secret.Name))
}

Python

import boto3
from botocore.exceptions import ClientError

# boto3 picks up credentials from the AWS credential provider chain:
# IRSA: AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE
# Pod Identity: handled transparently
# Local dev: ~/.aws/credentials
session = boto3.Session(region_name="us-east-1")

# S3 example
s3 = session.client("s3")
try:
    response = s3.get_object(Bucket="my-app-bucket", Key="config/settings.json")
    data = response["Body"].read().decode("utf-8")
    print(f"Retrieved S3 object: {len(data)} bytes")
except ClientError as e:
    print(f"Error: {e}")

# Secrets Manager example
sm = session.client("secretsmanager")
try:
    response = sm.get_secret_value(SecretId="my-app/database-password")
    print(f"Secret value: {response['SecretString']}")
except ClientError as e:
    print(f"Error: {e}")

# DynamoDB example
dynamodb = session.resource("dynamodb")
table = dynamodb.Table("my-table")
item = table.get_item(Key={"pk": "user-123"})
print(f"Item: {item.get('Item')}")

Node.js / TypeScript

import {
  S3Client,
  GetObjectCommand,
  PutObjectCommand,
} from "@aws-sdk/client-s3";
import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";

// AWS SDK v3 automatically handles IRSA and Pod Identity credentials
const s3 = new S3Client({ region: "us-east-1" });
const sm = new SecretsManagerClient({ region: "us-east-1" });

async function main() {
  // S3 example
  const s3Response = await s3.send(
    new GetObjectCommand({
      Bucket: "my-app-bucket",
      Key: "config/settings.json",
    })
  );
  console.log("S3 ContentType:", s3Response.ContentType);

  // Secrets Manager example
  const secretResponse = await sm.send(
    new GetSecretValueCommand({
      SecretId: "my-app/database-password",
    })
  );
  console.log("Secret retrieved:", secretResponse.Name);
}

main().catch(console.error);

---

IAM Policy Best Practices

Principle of Least Privilege

// BAD: Full S3 access
{
  "Effect": "Allow",
  "Action": "s3:*",
  "Resource": "*"
}

// GOOD: Minimal required actions, scoped to specific resource
{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject"
  ],
  "Resource": "arn:aws:s3:::my-app-bucket/uploads/*"
}

Condition Keys for IRSA

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71:aud": "sts.amazonaws.com",
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71:sub": "system:serviceaccount:my-app:my-app-sa"
        }
      }
    }
  ]
}

Use StringEquals not StringLike in trust policies. Wildcards (*) in the sub condition would allow any service account to assume the role.

Resource-Based Policies (S3 Bucket Policy)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/role-my-app-irsa"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-app-bucket/*",
      "Condition": {
        "StringEquals": {
          "s3:prefix": ["uploads/", "exports/"]
        }
      }
    }
  ]
}

Common Service-Specific Policies

// DynamoDB: Read specific table
{
  "Effect": "Allow",
  "Action": [
    "dynamodb:GetItem",
    "dynamodb:Query",
    "dynamodb:Scan"
  ],
  "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/my-table"
}

// SQS: Consume messages from a queue
{
  "Effect": "Allow",
  "Action": [
    "sqs:ReceiveMessage",
    "sqs:DeleteMessage",
    "sqs:GetQueueAttributes"
  ],
  "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
}

// Secrets Manager: Read specific secrets
{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-app/*"
}

// ECR: Pull images (for node instance profile, not pod)
{
  "Effect": "Allow",
  "Action": [
    "ecr:GetAuthorizationToken",
    "ecr:BatchCheckLayerAvailability",
    "ecr:GetDownloadUrlForLayer",
    "ecr:BatchGetImage"
  ],
  "Resource": "*"
}

---

Cross-Account Access

A common pattern is having workloads in an EKS cluster in Account A access resources in Account B.

Setup Cross-Account Access

In Account B — create a role that Account A can assume:

# Trust policy for Account B role (trusts Account A role)
cat > /tmp/cross-account-trust.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_A_ID:role/role-my-app-irsa"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "unique-external-id-12345"
        }
      }
    }
  ]
}
EOF

aws iam create-role \
  --role-name "role-cross-account-access" \
  --assume-role-policy-document file:///tmp/cross-account-trust.json \
  --profile account-b-profile

aws iam attach-role-policy \
  --role-name "role-cross-account-access" \
  --policy-arn "arn:aws:iam::ACCOUNT_B_ID:policy/s3-policy" \
  --profile account-b-profile

In Account A — grant the pod's role permission to assume the Account B role:

{
  "Effect": "Allow",
  "Action": "sts:AssumeRole",
  "Resource": "arn:aws:iam::ACCOUNT_B_ID:role/role-cross-account-access"
}

In application code — use STS to assume the cross-account role:

stsSvc := sts.NewFromConfig(cfg)
creds := stscreds.NewAssumeRoleProvider(stsSvc, 
    "arn:aws:iam::ACCOUNT_B_ID:role/role-cross-account-access",
    func(o *stscreds.AssumeRoleOptions) {
        o.ExternalID = aws.String("unique-external-id-12345")
    },
)

cfgB, _ := config.LoadDefaultConfig(ctx,
    config.WithRegion("us-east-1"),
    config.WithCredentialsProvider(creds),
)
s3AccountB := s3.NewFromConfig(cfgB)

---

Multi-Cluster and Multi-Environment Patterns

Pattern: One Role Per Service Per Environment

role-payment-service-dev
role-payment-service-staging
role-payment-service-prod

for ENV in dev staging prod; do
  aws iam create-role \
    --role-name "role-payment-service-${ENV}" \
    --assume-role-policy-document "$(cat <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER_FOR_$ENV}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_PROVIDER_FOR_$ENV}:aud": "sts.amazonaws.com",
        "${OIDC_PROVIDER_FOR_$ENV}:sub": "system:serviceaccount:payments:payment-service-sa"
      }
    }
  }]
}
EOF
)"
done

Managing with Terraform

# variables.tf
variable "cluster_oidc_issuer_url" {
  type = string
}

variable "environment" {
  type = string  # dev, staging, prod
}

# iam.tf
data "aws_iam_policy_document" "irsa_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [
        "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${local.oidc_provider}"
      ]
    }

    condition {
      test     = "StringEquals"
      variable = "${local.oidc_provider}:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "${local.oidc_provider}:sub"
      values   = ["system:serviceaccount:${var.namespace}:${var.service_account_name}"]
    }
  }
}

resource "aws_iam_role" "irsa" {
  name               = "role-${var.service_name}-${var.environment}"
  assume_role_policy = data.aws_iam_policy_document.irsa_trust.json
  
  tags = {
    Environment = var.environment
    Service     = var.service_name
    ManagedBy   = "terraform"
  }
}

resource "aws_iam_role_policy_attachment" "irsa" {
  role       = aws_iam_role.irsa.name
  policy_arn = aws_iam_policy.service.arn
}

# Pod Identity alternative
resource "aws_eks_pod_identity_association" "main" {
  cluster_name    = var.cluster_name
  namespace       = var.namespace
  service_account = var.service_account_name
  role_arn        = aws_iam_role.pod_identity.arn
}

---

Security Best Practices

1. Never Use StringLike with Wildcards in Trust Policies

// BAD: Any service account in any namespace of this cluster can assume this role
"Condition": {
  "StringLike": {
    "${OIDC_PROVIDER}:sub": "system:serviceaccount:*:*"
  }
}

// GOOD: Only this specific service account
"Condition": {
  "StringEquals": {
    "${OIDC_PROVIDER}:sub": "system:serviceaccount:my-app:my-app-sa"
  }
}

2. Enable CloudTrail for IAM Role Activity

aws cloudtrail create-trail \
  --name "eks-iam-audit" \
  --s3-bucket-name "my-audit-logs-bucket" \
  --include-global-service-events \
  --is-multi-region-trail \
  --enable-log-file-validation

# CloudTrail captures AssumeRoleWithWebIdentity calls
# showing exactly which pod/serviceaccount assumed which role

3. Use AWS IAM Access Analyzer

# Scan for roles with overly broad access
aws accessanalyzer create-analyzer \
  --analyzer-name "eks-irsa-analyzer" \
  --type ACCOUNT

aws accessanalyzer list-findings \
  --analyzer-name "eks-irsa-analyzer"

4. Restrict the OIDC Audience

// Always require sts.amazonaws.com as audience
"Condition": {
  "StringEquals": {
    "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
  }
}

5. Set Maximum Session Duration Appropriately

aws iam update-role \
  --role-name "role-my-app-irsa" \
  --max-session-duration 3600  # 1 hour; default is 3600, max is 43200

6. Use AWS Secrets Manager Instead of Kubernetes Secrets for Sensitive Values

# Use External Secrets Operator to sync Secrets Manager → Kubernetes Secrets
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: my-app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: my-app/database
        property: password

---

Troubleshooting

WebIdentityErr: Failed to Retrieve Credentials

# Check if the token file exists in the pod
kubectl exec -n my-app deployment/my-app -- \
  ls -la /var/run/secrets/eks.amazonaws.com/serviceaccount/

# Verify environment variables are present
kubectl exec -n my-app deployment/my-app -- env | grep AWS

# Check the token is not expired (default 24h)
kubectl exec -n my-app deployment/my-app -- \
  cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token | \
  cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.tool | grep -E "exp|iss|sub"

AccessDenied When Calling AWS Service

# Test the role assumption manually
aws sts assume-role-with-web-identity \
  --role-arn $ROLE_ARN \
  --role-session-name "test-session" \
  --web-identity-token "$(kubectl exec -n my-app deployment/my-app -- \
    cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token)"

# If assumption succeeds, test the actual service call with assumed creds
# If assumption fails, the trust policy is misconfigured

# Check the trust policy subjects match the actual KSA
kubectl get serviceaccount my-app-sa -n my-app -o yaml
# Compare the namespace+name against the sub in the trust policy

IRSA: Role Not Being Assumed (No Annotation Effect)

# Verify the Pod Identity Webhook is running
kubectl get pods -n kube-system | grep pod-identity-webhook

# Check pod's service account annotation was applied
kubectl get serviceaccount my-app-sa -n my-app -o yaml | grep role-arn

# Verify the pod's service account matches the annotated one
kubectl get pod <pod-name> -n my-app -o yaml | grep serviceAccountName

EKS Pod Identity: Credentials Not Available

# Verify Pod Identity Agent DaemonSet is running on every node
kubectl get daemonset eks-pod-identity-agent -n kube-system
kubectl get pods -n kube-system -l app.kubernetes.io/name=eks-pod-identity-agent

# Verify the association exists
aws eks list-pod-identity-associations \
  --cluster-name $CLUSTER_NAME \
  --region $REGION \
  --namespace my-app

# Describe the association
aws eks describe-pod-identity-association \
  --cluster-name $CLUSTER_NAME \
  --region $REGION \
  --association-id <association-id>

Validate IAM Permissions (Without Deploying)

# Simulate policy evaluation
aws iam simulate-principal-policy \
  --policy-source-arn "arn:aws:iam::${ACCOUNT_ID}:role/role-my-app-irsa" \
  --action-names "s3:GetObject" \
  --resource-arns "arn:aws:s3:::my-app-bucket/uploads/test.json"

---

What I Learned

After migrating multiple EKS clusters from static credentials to IRSA, and more recently to Pod Identity, here are the lessons I keep coming back to:

1. Start with Pod Identity for new clusters. IRSA is battle-tested and widely supported in Terraform modules, but Pod Identity is simpler to manage at scale — no cluster-specific OIDC provider references in trust policies means you can reuse roles across clusters without updating policies.

2. Always use StringEquals in trust policy conditions, never StringLike with wildcards. A wildcarded trust policy is effectively giving every workload in your cluster access to that role's permissions.

3. One IAM role per service per environment. Shared roles mean shared blast radius. The overhead of maintaining separate roles pays for itself when an incident isolates to one service rather than cascading.

4. LoadDefaultConfig / boto3.Session() just works. Don't implement custom credential providers. The AWS SDK credential chain handles IRSA, Pod Identity, instance profiles, and local ~/.aws/credentials transparently — same code runs everywhere.

5. CloudTrail is your audit trail. Every AssumeRoleWithWebIdentity call shows the source OIDC token's subject (system:serviceaccount:<ns>:<name>). Enable CloudTrail from day one; it's invaluable for debugging AND compliance.

6. ECR image pulls use the node instance profile, not IRSA. Don't annotate your service account with ECR permissions expecting it to help with image pulls — the kubelet on the node handles that via its own IAM role.

7. Test your trust policy before deploying. aws iam simulate-principal-policy saves deployment debugging cycles. Paste your trust policy into the IAM Policy Simulator to confirm conditions work as expected.

---

Summary

Next steps:

• Kubernetes 101: MS Entra Managed Identity in Kubernetes

• Kubernetes 101: Configuration and Secrets

• Kubernetes 101: Namespaces and RBAC

• Kubernetes 101: Production Best Practices