ArgoCD Architecture and Components

When I Needed to Understand ArgoCD Internals

Our ArgoCD instance was syncing 127 applications across 4 Kubernetes clusters. Everything worked perfectly... until one day it didn't.

# Check ArgoCD status
kubectl get pods -n argocd
# argocd-server: CrashLoopBackOff
# argocd-repo-server: Running
# argocd-application-controller: Running but CPU 400%

# Check logs
kubectl logs -n argocd argocd-application-controller
# [ERROR] Failed to sync application: timeout
# [ERROR] Failed to sync application: timeout
# [ERROR] Failed to sync application: timeout
# ... 127 timeouts

The application controller was overwhelmed. But why? What was it actually doing?

I needed to understand ArgoCD's architecture to fix this. Let me share what I learned.

ArgoCD High-Level Architecture

ArgoCD is a Kubernetes-native continuous delivery tool. It runs inside your Kubernetes cluster as a set of pods.

Core components:

API Server - Web UI, CLI, API endpoints
Repository Server - Fetches manifests from Git
Application Controller - Reconciliation loop (sync Git → cluster)
Redis - Cache and message queue
Dex - SSO integration (optional)

Let's dive into each component.

API Server (argocd-server)

The API server is the frontend of ArgoCD.

What It Does

Responsibilities:

Serve Web UI (port 8080)
Handle CLI requests (port 8080)
Expose REST/gRPC API (port 8080, 8083)
Authentication (local users, SSO via Dex)
Authorization (RBAC)
Proxy to Kubernetes API
Cache reads from Redis

API Server Pod

kubectl get pod -n argocd -l app.kubernetes.io/name=argocd-server
# NAME                            READY   STATUS
# argocd-server-7d9c8c4c5-abc12   1/1     Running

kubectl describe pod -n argocd argocd-server-7d9c8c4c5-abc12
# Container: argocd-server
# Image: quay.io/argoproj/argocd:v2.9.3
# Ports: 8080 (HTTP), 8083 (gRPC)
# Args:
#   --staticassets /shared/app
#   --repo-server argocd-repo-server:8081
#   --redis argocd-redis:6379

What I Learned: API Server Scaling

When you have 100+ applications and 50+ users, you need to scale the API server.

# Scale API server for high traffic
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
  namespace: argocd
spec:
  replicas: 3  # Default: 1, we scaled to 3
  template:
    spec:
      containers:
      - name: argocd-server
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi

Signs you need to scale:

UI is slow
CLI commands timeout
High CPU on argocd-server pod
Many users accessing simultaneously

Repository Server (argocd-repo-server)

The repo server is the Git interface of ArgoCD.

What It Does

Responsibilities:

Clone Git repositories
Fetch manifests from Git
Run Kustomize build
Render Helm charts
Execute config management plugins
Cache results in Redis
Serve manifests to application controller

Repo Server Pod

kubectl get pod -n argocd -l app.kubernetes.io/name=argocd-repo-server
# NAME                                  READY   STATUS
# argocd-repo-server-6b8c5f9d7-xyz89    1/1     Running

kubectl exec -it argocd-repo-server-6b8c5f9d7-xyz89 -n argocd -- ls /tmp
# argocd-local-821932719  ← Cloned repos cached here
# hs-cache                ← Helm/Kustomize build cache

What I Learned: Repo Server is CPU-Intensive

Rendering Helm charts and building Kustomize manifests is expensive.

Problem I faced:

127 applications
50+ Helm charts
Sync every 3 minutes
Repo server: CPU 300-400%

Solution:

# Scale repo server horizontally
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
  namespace: argocd
spec:
  replicas: 3  # Default: 1, we scaled to 3
  template:
    spec:
      containers:
      - name: argocd-repo-server
        resources:
          requests:
            cpu: 500m      # Increased from 100m
            memory: 512Mi  # Increased from 256Mi
          limits:
            cpu: 2000m
            memory: 2Gi
        env:
        # Tune cache size
        - name: ARGOCD_EXEC_TIMEOUT
          value: "180s"  # Default: 90s

Signs you need to scale:

Sync timeouts
High CPU on repo-server pod
Slow manifest generation
Many Helm/Kustomize apps

Application Controller (argocd-application-controller)

The application controller is the heart of ArgoCD. This is where GitOps reconciliation happens.

What It Does

Responsibilities:

Watch Application CRDs
Fetch desired state from Git (via repo server)
Fetch actual state from Kubernetes
Compare desired vs actual
Sync if different (apply manifests)
Monitor application health
Detect drift
Update application status
Trigger sync hooks and waves

Application Controller Pod

kubectl get pod -n argocd -l app.kubernetes.io/name=argocd-application-controller
# NAME                                            READY   STATUS
# argocd-application-controller-0                 1/1     Running

# This is a StatefulSet (not Deployment)
kubectl get statefulset -n argocd
# NAME                            READY
# argocd-application-controller   1/1

Why StatefulSet?

Need stable identity
Sharding support (multi-replica)
Each replica handles subset of applications

What I Learned: Application Controller is the Bottleneck

With 127 applications syncing every 3 minutes, the controller was overwhelmed.

Math:

127 applications × 3-minute sync interval = 42 syncs/minute
Each sync:
  - Fetch from Git: ~2 seconds
  - Compare with cluster: ~1 second
  - Apply if needed: ~3 seconds
  - Update status: ~1 second
  Total: ~7 seconds/app

42 apps/min × 7 sec = 294 seconds = 5 minutes
But we only have 3 minutes! = QUEUE BUILDUP

Solution: Enable sharding

# Enable horizontal scaling with sharding
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: argocd-application-controller
  namespace: argocd
spec:
  replicas: 3  # Run 3 replicas
  template:
    spec:
      containers:
      - name: argocd-application-controller
        env:
        # Enable sharding
        - name: ARGOCD_CONTROLLER_REPLICAS
          value: "3"
        - name: ARGOCD_CONTROLLER_SHARD_NUMBER
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        resources:
          requests:
            cpu: 1000m
            memory: 1Gi
          limits:
            cpu: 2000m
            memory: 2Gi

How sharding works:

Controller 0: Applications 0, 3, 6, 9, ... (mod 3 = 0)
Controller 1: Applications 1, 4, 7, 10, ... (mod 3 = 1)
Controller 2: Applications 2, 5, 8, 11, ... (mod 3 = 2)

127 apps ÷ 3 controllers = ~42 apps each
42 apps × 7 sec = 294 sec = under 5 min per controller

Redis

Redis is the cache and message bus for ArgoCD.

What It Stores

Data stored:

Git repository cache (clone results)
Manifest generation cache (Helm/Kustomize output)
Application state cache
Sync operation queue
User session data
RBAC policy cache

Redis Pod

kubectl get pod -n argocd -l app.kubernetes.io/name=argocd-redis
# NAME                     READY   STATUS
# argocd-redis-74c9...     1/1     Running

# Check Redis data
kubectl exec -it argocd-redis-74c9... -n argocd -- redis-cli
127.0.0.1:6379> KEYS *
1) "cache|app|my-app|state"
2) "cache|repo|https://github.com/user/repo|abc123|manifests"
3) "session|user|alice|token"
...

What I Learned: Redis Can Be a Bottleneck

Problem: With many apps and frequent syncs, Redis can hit memory limits.

Solution:

# Increase Redis memory
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-redis
  namespace: argocd
spec:
  template:
    spec:
      containers:
      - name: redis
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 200m
            memory: 512Mi  # Increased from 256Mi
        args:
        - --maxmemory
        - 512mb
        - --maxmemory-policy
        - allkeys-lru  # Evict least recently used keys

For production with 100+ apps:

Use Redis with persistence (enable AOF)
Or use external Redis (AWS ElastiCache, etc.)
Monitor Redis memory usage

Dex (Optional SSO Server)

Dex is an identity provider that integrates with SSO systems.

What It Does

Supports:

OIDC (OpenID Connect)
SAML 2.0
LDAP
GitHub
Google
GitLab
Okta
Azure AD
etc.

Dex Configuration Example

# ConfigMap: argocd-cm
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.example.com
  dex.config: |
    connectors:
    - type: github
      id: github
      name: GitHub
      config:
        clientID: $GITHUB_CLIENT_ID
        clientSecret: $GITHUB_CLIENT_SECRET
        orgs:
        - name: my-company
          teams:
          - platform-team

Without Dex:

Local user accounts only
Manually manage users/passwords
No SSO integration

With Dex:

SSO with corporate identity provider
Automatic user provisioning
Group/team-based access

How Components Work Together

Let's trace a complete sync operation:

Scenario: Developer Pushes to Git

# Developer updates deployment
git commit -m "Update API to v2.3.0"
git push origin main

Step-by-Step Flow

1. Application Controller polls Git (every 3 min)

Application Controller:
  "Time to check my-app..."
  → Request to Repo Server: "Get manifests for my-app"

2. Repo Server fetches from Git

Repo Server:
  → Check Redis cache: "Do I have this commit?"
  → Cache miss
  → Clone/pull from Git
  → Run Kustomize/Helm build
  → Store in Redis cache
  → Return manifests to Application Controller

3. Application Controller compares states

Application Controller:
  → Get desired state (from Repo Server)
  → Get actual state (from Kubernetes API)
  → Compare:
      Desired: image=api:v2.3.0
      Actual:  image=api:v2.2.0
      Status: OUT OF SYNC

4. Application Controller syncs

Application Controller:
  → Apply manifest to Kubernetes
  → kubectl apply -f deployment.yaml (internally)
  → Wait for rollout
  → Check health status
  → Update Application CRD status: Synced ✓

5. API Server shows status

API Server:
  → User opens Web UI
  → Fetch app status from Application Controller
  → Show: "my-app: Synced, Healthy"

Mermaid Sequence Diagram

Component Communication

Ports:

API Server: 8080 (HTTP), 8083 (gRPC)
Repo Server: 8081 (gRPC)
Redis: 6379 (TCP)
Dex: 5556 (HTTP)
Kubernetes API: 6443 (HTTPS)

Performance Tuning Based on Architecture

For 1-50 Applications

# Default setup works fine
API Server: 1 replica
Repo Server: 1 replica
Application Controller: 1 replica
Redis: 1 replica (256Mi memory)

For 50-200 Applications

# Scale horizontally
API Server: 2-3 replicas
Repo Server: 2-3 replicas (increase CPU/memory)
Application Controller: 2-3 replicas (enable sharding)
Redis: 1 replica (512Mi memory)

For 200+ Applications

# Heavy scaling
API Server: 3-5 replicas
Repo Server: 3-5 replicas (2 CPU, 2Gi memory each)
Application Controller: 3-5 replicas (sharding, 2 CPU, 2Gi memory each)
Redis: External managed service (ElastiCache, etc.)

Troubleshooting Based on Architecture

Slow UI/CLI

Problem: API Server overwhelmed Check:

kubectl top pod -n argocd -l app.kubernetes.io/name=argocd-server

Solution: Scale API server replicas

Sync Timeouts

Problem: Repo Server slow or Application Controller overwhelmed Check:

kubectl top pod -n argocd -l app.kubernetes.io/name=argocd-repo-server
kubectl top pod -n argocd -l app.kubernetes.io/name=argocd-application-controller

Solution:

Scale Repo Server (if high CPU)
Scale Application Controller with sharding (if many apps)

High Memory Usage

Problem: Redis cache too small Check:

kubectl exec -it argocd-redis-xxx -n argocd -- redis-cli INFO memory

Solution: Increase Redis memory limits

Key Takeaways

ArgoCD has 5 core components
- API Server: Frontend (UI, CLI, API)
- Repo Server: Git interface (clone, build manifests)
- Application Controller: Reconciliation engine (sync Git → cluster)
- Redis: Cache and message bus
- Dex: SSO integration (optional)
Application Controller is the heart
- Does the actual GitOps reconciliation
- Watches Application CRDs
- Compares desired (Git) vs actual (cluster)
- Applies changes
Repo Server is CPU-intensive
- Renders Helm charts
- Builds Kustomize manifests
- Scale when you have many Helm/Kustomize apps
Scale based on load
- 1-50 apps: Default setup
- 50-200 apps: Scale to 2-3 replicas
- 200+ apps: Scale to 3-5 replicas, enable sharding
Understand the flow
- Dev pushes → Git
- Controller polls → Repo Server
- Repo Server fetches → Git
- Controller compares → desired vs actual
- Controller syncs → Kubernetes

In the next article, we'll get hands-on: installing ArgoCD on a Kubernetes cluster, configuring it, and preparing it for your first GitOps deployment.

Previous: Understanding GitOps Core Concepts Next: Installing and Configuring ArgoCD

PreviousUnderstanding GitOps Core Concepts NextInstalling and Configuring ArgoCD

Last updated 1 month ago

hashtagWhen I Needed to Understand ArgoCD Internals

hashtagArgoCD High-Level Architecture

hashtagAPI Server (argocd-server)

hashtagWhat It Does

hashtagAPI Server Pod

hashtagWhat I Learned: API Server Scaling

hashtagRepository Server (argocd-repo-server)

hashtagWhat It Does

hashtagRepo Server Pod

hashtagWhat I Learned: Repo Server is CPU-Intensive

hashtagApplication Controller (argocd-application-controller)

hashtagWhat It Does

hashtagApplication Controller Pod

hashtagWhat I Learned: Application Controller is the Bottleneck

hashtagRedis

hashtagWhat It Stores

hashtagRedis Pod

hashtagWhat I Learned: Redis Can Be a Bottleneck

hashtagDex (Optional SSO Server)

hashtagWhat It Does

hashtagDex Configuration Example

hashtagHow Components Work Together

hashtagScenario: Developer Pushes to Git

hashtagStep-by-Step Flow

hashtagMermaid Sequence Diagram

hashtagComponent Communication

hashtagPerformance Tuning Based on Architecture

hashtagFor 1-50 Applications

hashtagFor 50-200 Applications

hashtagFor 200+ Applications

hashtagTroubleshooting Based on Architecture

hashtagSlow UI/CLI

hashtagSync Timeouts

hashtagHigh Memory Usage

hashtagKey Takeaways

When I Needed to Understand ArgoCD Internals

ArgoCD High-Level Architecture

API Server (argocd-server)

What It Does

API Server Pod

What I Learned: API Server Scaling

Repository Server (argocd-repo-server)

What It Does

Repo Server Pod

What I Learned: Repo Server is CPU-Intensive

Application Controller (argocd-application-controller)

What It Does

Application Controller Pod

What I Learned: Application Controller is the Bottleneck

Redis

What It Stores

Redis Pod

What I Learned: Redis Can Be a Bottleneck

Dex (Optional SSO Server)

What It Does

Dex Configuration Example

How Components Work Together

Scenario: Developer Pushes to Git

Step-by-Step Flow

Mermaid Sequence Diagram

Component Communication

Performance Tuning Based on Architecture

For 1-50 Applications

For 50-200 Applications

For 200+ Applications

Troubleshooting Based on Architecture

Slow UI/CLI

Sync Timeouts

High Memory Usage

Key Takeaways