Automation Mesh and Execution Environments

Scaling from 50 to 1000 Nodes Across Three Regions

When our infrastructure grew from a single datacenter to three regions (US East, Europe, Asia Pacific), our centralized AAP controller struggled. Network latency caused job timeouts. Bandwidth costs exploded. Regional compliance requirements prohibited cross-border automation traffic.

Traditional solution: Deploy separate AAP instances per region. Problem: Three separate platforms to manage, no centralized visibility, credential sync nightmares.

Automation Mesh solved everything. One control plane, distributed execution nodes, intelligent routing, regional isolation. Job execution stayed local, management stayed centralized.

Result: 70% latency reduction, 60% bandwidth savings, single-pane-of-glass management.

What You'll Learn

• Automation Mesh architecture and topology patterns

• Multi-region deployment strategies

• Execution Environments (container-based execution)

• Building custom Execution Environments

• ansible-builder usage

• Performance tuning and capacity planning

• Troubleshooting mesh connectivity

Automation Mesh Deep Dive

Mesh Topology Patterns

Hub-and-Spoke (Most Common)

Topology:
  Control Plane (HQ):
    - controller-01.example.com
    - controller-02.example.com
  
  Region: US-East (AWS)
    Hop Node: hop-use1.example.com
    Execution Nodes:
      - exec-use1-01 through exec-use1-10
  
  Region: EU-West (Azure)
    Hop Node: hop-euw1.example.com
    Execution Nodes:
      - exec-euw1-01 through exec-euw1-05
  
  Region: AP-Southeast (GCP)
    Hop Node: hop-apse1.example.com
    Execution Nodes:
      - exec-apse1-01 through exec-apse1-05

Benefits:

• Jobs execute close to managed infrastructure

• Reduced latency and bandwidth

• Simplified firewall rules

• Regional compliance (data stays in region)

Mesh Deployment

Installation inventory file:

[automationcontroller]
controller-01.example.com
controller-02.example.com

[automationcontroller:vars]
peers=execution_nodes

# Hop nodes - regional gateways
[execution_nodes]
hop-use1.example.com receptor_type=hop node_type=hop
hop-euw1.example.com receptor_type=hop node_type=hop

# Execution nodes - run jobs
exec-use1-[01:10].example.com receptor_type=execution node_type=execution
exec-euw1-[01:05].example.com receptor_type=execution node_type=execution

# Mesh connectivity
[execution_nodes:vars]
peers=automationcontroller

Real-world mesh topology:

Our Production Mesh:
  Control Nodes: 3 (HA cluster)
  Hop Nodes: 3 (one per region)
  Execution Nodes: 30 (distributed)
  
  Traffic Flow:
    Controller → Hop (encrypted, multiplexed)
    Hop → Execution Nodes (regional, low latency)
    Execution → Managed Infrastructure (same region)

Instance Groups

Organize execution nodes into groups:

Instance Groups:
  aws-us-east-1:
    instances:
      - exec-use1-01
      - exec-use1-02
    policy: AWS US-East workloads
  
  azure-eu-west-1:
    instances:
      - exec-euw1-01
      - exec-euw1-02
    policy: Azure Europe workloads
  
  on-prem-datacenter:
    instances:
      - exec-onprem-01
    policy: On-premises infrastructure

# Assign to job templates
Job Template "Deploy AWS App":
  Instance Groups: [aws-us-east-1]
  
# Jobs run on geographically close execution nodes

Execution Environments

What Are Execution Environments?

Container images containing:

• Ansible Core

• Collections

• Python dependencies

• System libraries

• Custom tools

Benefits:

• Consistent execution environment

• Version pinning

• Isolation between jobs

• Easy distribution

• Rapid deployment

Using Pre-Built EEs

# From Automation Hub or registry.redhat.io
Execution Environments:
  ee-minimal-rhel8:
    description: Minimal Ansible 2.15
    use_case: Simple playbooks
  
  ee-supported-rhel8:
    description: Supported collections
    use_case: General automation
    includes:
      - ansible.posix
      - ansible.utils
      - ansible.windows
  
  ee-29-rhel8:
    description: Ansible 2.9 for legacy
    use_case: Legacy playbook compatibility

Building Custom Execution Environments

Use Case: Network automation requiring napalm, netmiko, ncclient

Step 1: Define Requirements

execution-environment.yml:

version: 3

images:
  base_image:
    name: registry.redhat.io/ansible-automation-platform-24/ee-minimal-rhel8:latest

dependencies:
  galaxy: requirements.yml
  python: requirements.txt
  system: bindep.txt

additional_build_steps:
  prepend_galaxy:
    - RUN pip3 install --upgrade pip setuptools
  
  append_final:
    - RUN echo "Custom network automation EE" > /etc/motd

requirements.yml (Ansible collections):

collections:
  - name: ansible.netcommon
    version: ">=5.0.0"
  
  - name: cisco.ios
    version: ">=4.0.0"
  
  - name: junipernetworks.junos
    version: ">=5.0.0"
  
  - name: arista.eos
    version: ">=6.0.0"

requirements.txt (Python packages):

napalm>=4.0.0
netmiko>=4.1.0
ncclient>=0.6.13
paramiko>=3.0.0
textfsm>=1.1.3

bindep.txt (System packages):

python38-devel [platform:rpm]
gcc [platform:rpm]
libxml2-devel [platform:rpm]
libxslt-devel [platform:rpm]

Step 2: Build with ansible-builder

# Install ansible-builder
pip install ansible-builder

# Build the EE
ansible-builder build \
  --tag network-automation-ee:1.0 \
  --container-runtime podman \
  --verbosity 3

# Push to registry
podman push network-automation-ee:1.0 \
  hub.example.com/network-automation-ee:1.0

Step 3: Register in AAP

# Via UI: Administration → Execution Environments → Add
Name: Network Automation EE
Image: hub.example.com/network-automation-ee:1.0
Pull: Always
Credential: Container-Registry-Auth

# Via API
curl -X POST https://controller.example.com/api/v2/execution_environments/ \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "name": "Network Automation EE",
    "image": "hub.example.com/network-automation-ee:1.0",
    "pull": "always"
  }'

Step 4: Use in Job Template

Job Template:
  Name: "Configure Network Devices"
  Execution Environment: Network Automation EE
  Project: Network Playbooks
  Playbook: configure_switches.yml
  
# Job runs in this specific EE with all dependencies

Real-world EE strategy:

Our Custom EEs:
  base-ee:1.0
    - Common collections
    - Standard tools
    - Base for other EEs
  
  aws-automation-ee:2.0
    - Base EE +
    - AWS collections
    - boto3, botocore
  
  azure-automation-ee:1.5
    - Base EE +
    - Azure collections
    - Azure SDK
  
  network-automation-ee:3.0
    - Base EE +
    - Network collections
    - napalm, netmiko
  
  database-automation-ee:1.2
    - Base EE +
    - psycopg2, pymysql
    - Database collections

Performance Tuning

Execution Capacity

Control Node Capacity:
  Total = (CPU cores × 10) - 10
  Example: 8 cores = (8 × 10) - 10 = 70 forks
  
Execution Node Capacity:
  Total = (CPU cores × 10)
  Example: 4 cores = 4 × 10 = 40 forks

Capacity Management:
  - Monitor capacity usage
  - Add execution nodes when >80% used
  - Use instance groups for workload distribution

Mesh Performance

Optimize:
  - Place execution nodes close to managed infrastructure
  - Use hop nodes to reduce WAN traffic
  - Enable compression for slow links
  - Monitor receptor mesh status
  
Network Requirements:
  Control ↔ Hop: Stable, low latency
  Hop ↔ Execution: Regional, high bandwidth
  Execution ↔ Managed: Minimal latency

Bandwidth Savings:
  Before mesh: All traffic through controller
  After mesh: Regional traffic stays regional
  Result: 60% bandwidth reduction in our case

Troubleshooting

Mesh Connectivity Issues

# Check receptor status
receptorctl status

# Test connectivity between nodes
receptorctl ping <node-name>

# Check routes
receptorctl traceroute <node-name>

# Verify node mesh connections
awx-manage list_instances

Execution Environment Issues

# Test EE locally
podman run -it network-automation-ee:1.0 /bin/bash
ansible --version
ansible-galaxy collection list

# Check EE pull logs
podman logs <container-id>

# Verify registry authentication
podman login hub.example.com

Common Issues

Issue: Jobs not running on expected execution nodes

Solution:
  - Check instance group assignment
  - Verify node capacity
  - Check node status (online/offline)
  - Review instance group policy

Issue: EE image pull failures

Solution:
  - Verify registry credentials
  - Check network connectivity
  - Confirm image tag exists
  - Review pull policy settings

Key Takeaways

✅ Automation Mesh enables multi-region scaling ✅ Hub-and-spoke topology for most deployments ✅ Execution Environments provide consistent execution ✅ ansible-builder creates custom EEs easily ✅ Instance groups route workloads intelligently ✅ Performance improves with proper topology

What's Next

The next article introduces Event-Driven Ansible - building real-time, reactive automation that responds to events automatically.

---

Next Article: Introduction to Event-Driven Ansible →

---

Part of the Ansible Automation Platform 101 Series