Credentials Management

The Day We Rotated 200+ Credentials Without Breaking Anything

Our security team mandated credential rotation every 90 days. We had 200+ credentials spread across AAP: SSH keys for 150 servers, 20 cloud accounts, 15 Git repositories, 10 database passwords, and various API tokens.

The first rotation attempt was a disaster. We manually updated credentials in AAP, but missed updating vault files, broke half our playbooks, and spent 2 days firefighting.

Then I learned about AAP's credentials management and external vault integration. We integrated with HashiCorp Vault, configured automatic credential sync, implemented dynamic credential injection, and built automated rotation workflows.

Next rotation: Zero downtime, zero manual updates, complete success.

What You'll Learn

• Custom credential types for any authentication

• HashiCorp Vault and CyberArk integration

• Dynamic credential injection patterns

• Credential rotation strategies

• Cloud credential management

• Security best practices

• Compliance and auditing

Custom Credential Types

Creating Custom Types

Example: API Token Credential

Name: Custom API Token
Kind: Cloud

Input Configuration:
  fields:
    - id: api_url
      type: string
      label: API URL
      help_text: Base URL for API
      
    - id: api_token
      type: string
      label: API Token
      secret: true
      help_text: Authentication token
      
    - id: api_version
      type: string
      label: API Version
      default: "v2"
  
  required:
    - api_url
    - api_token

Injector Configuration:
  env:
    API_URL: '{{ api_url }}'
    API_TOKEN: '{{ api_token }}'
    API_VERSION: '{{ api_version }}'
  
  extra_vars:
    api_endpoint: '{{ api_url }}/{{ api_version }}'
    api_auth_header: 'Bearer {{ api_token }}'

Real-world use: Custom types for internal APIs, vendor-specific authentication, proprietary systems.

Example: Database Credential

Name: PostgreSQL Database
Kind: Cloud

Input:
  fields:
    - id: db_host
      type: string
      label: Database Host
    
    - id: db_port
      type: string
      label: Port
      default: "5432"
    
    - id: db_name
      type: string
      label: Database Name
    
    - id: db_username
      type: string
      label: Username
    
    - id: db_password
      type: string
      label: Password
      secret: true
    
    - id: db_ssl_mode
      type: string
      label: SSL Mode
      choices: ["disable", "require", "verify-ca", "verify-full"]

Injector:
  extra_vars:
    postgres_connection_string: 'postgresql://{{ db_username }}:{{ db_password }}@{{ db_host }}:{{ db_port }}/{{ db_name }}?sslmode={{ db_ssl_mode }}'

HashiCorp Vault Integration

Configuration

# In AAP: Settings → HashiCorp Vault

Vault URL: https://vault.example.com:8200
Token: (Vault authentication token)
API Version: v2

# Or use AppRole authentication
AppRole ID: (AppRole ID)
Secret ID: (Secret ID)

Using Vault-Backed Credentials

Create Credential with Vault Lookup:

Credential Type: HashiCorp Vault Secret Lookup
Name: AWS-Production-Keys

Vault Path: secret/data/aws/production
Vault Key: access_key_id  # For AWS Access Key
Vault Secret: secret_access_key  # For AWS Secret Key

# AAP fetches from Vault at job runtime
# Secrets never stored in AAP database

Playbook Usage:

# Credential automatically injected
- name: Use AWS credential
  amazon.aws.ec2_instance:
    # AWS keys from Vault via credential
    region: us-east-1
    name: web-server

Vault Dynamic Secrets

Pattern: Generate credentials on-demand

# Vault configured for dynamic database credentials
Vault Path: database/creds/readonly

When job runs:
  1. AAP requests credential from Vault
  2. Vault generates temporary database user
  3. Credential injected into playbook
  4. After job: Vault revokes credential

Benefits:
  - No static credentials
  - Auto-expiring access
  - Full audit trail

Real-world implementation:

Our Vault Integration:
  - AWS credentials: Dynamic, 1-hour TTL
  - Database passwords: Dynamic, 30-minute TTL
  - API tokens: Static, synced hourly
  - SSH keys: Static, rotated quarterly
  - Certificates: Dynamic, auto-renewed

CyberArk Integration

Configuration

Credential Type: CyberArk Central Credential Provider Lookup
Name: Production-SSH-Keys

CyberArk URL: https://cyberark.example.com/AIMWebService
Application ID: AAP-Production
Safe: SSH-Keys
Object: prod-ansible-user

Features:

• Check-in/check-out workflows

• Password rotation by CyberArk

• Privileged access management

• Compliance reporting

Cloud Credential Management

AWS Credentials

Static Keys:

Credential Type: Amazon Web Services
Name: AWS-Production

Access Key: AKIAIOSFODNN7EXAMPLE
Secret Key: (encrypted in AAP)

Usage:
  - EC2 inventory
  - AWS modules in playbooks
  - CloudFormation deployments

AssumeRole Pattern:

# Use base credential to assume different roles

Base Credential: AWS-OrganizationAccount
IAM Role ARN: arn:aws:iam::123456789012:role/AnsibleAutomation

Benefits:
  - Single credential, multiple accounts
  - Temporary credentials
  - Better audit trail

Azure Credentials

Credential Type: Microsoft Azure Resource Manager

Subscription ID: xxx-xxx-xxx
Tenant ID: yyy-yyy-yyy
Client ID: zzz-zzz-zzz
Client Secret: (encrypted)

# Or use Managed Identity (recommended)
Use Managed Identity: Yes
Subscription ID: xxx-xxx-xxx

GCP Credentials

Credential Type: Google Compute Engine

Service Account Email: [email protected]
Service Account JSON: (upload JSON key file)

# Or use Workload Identity (GKE)

Credential Rotation Strategies

Pattern 1: Scheduled Rotation

Automation:
  1. Scheduled workflow runs weekly
  2. Generates new credentials
  3. Updates target systems
  4. Updates AAP credentials
  5. Verifies with test job
  6. Notifies on completion/failure

Example Playbook:
  - Generate new SSH key pair
  - Deploy public key to servers
  - Update AAP credential via API
  - Test with simple job
  - Archive old key

Pattern 2: Vault-Based Auto-Rotation

Using HashiCorp Vault:
  - Vault owns credential lifecycle
  - AAP always fetches latest
  - No AAP updates needed
  
Vault Rotation Policy:
  - AWS: Rotate every 30 days
  - Database: Rotate every 7 days
  - API tokens: Rotate every 60 days

Pattern 3: External Trigger Rotation

Triggered by:
  - Security incident
  - Employee departure
  - Compliance requirement
  
Process:
  1. Webhook triggers AAP workflow
  2. Workflow rotates all affected credentials
  3. Updates all systems
  4. Notifies security team
  5. Creates audit report

Real-world rotation schedule:

SSH Keys: 90 days
Cloud Credentials: 60 days
Database Passwords: 30 days
API Tokens: 90 days
Certificates: 365 days (Let's Encrypt auto-renews)
Emergency: Immediate on security event

Credential Security Best Practices

Principle of Least Privilege

Wrong:
  Credential: AWS-Admin-FullAccess
  Permissions: AdministratorAccess
  Used by: Everyone

Right:
  Credential: AWS-EC2-ReadOnly
  Permissions: EC2:Describe*, EC2:List*
  Used by: Inventory sync only
  
  Credential: AWS-EC2-Operations
  Permissions: EC2:Start*, EC2:Stop*, EC2:Reboot*
  Used by: Operations team
  
  Credential: AWS-EC2-Deployment
  Permissions: EC2:RunInstances, EC2:TerminateInstances
  Used by: Deployment workflows

Credential Segregation

Development:
  - Dev-SSH-Keys (non-production servers)
  - Dev-AWS-Credentials (sandbox account)
  - Dev-Database-Passwords (dev databases)

Production:
  - Prod-SSH-Keys (production servers)
  - Prod-AWS-Credentials (production account)
  - Prod-Database-Passwords (production databases)
  - NO access from Dev teams

Audit and Monitoring

Monitor:
  - Credential usage frequency
  - Failed authentication attempts
  - Credential modifications
  - Permission grants/revokes
  - External vault access

Alerts:
  - Credential used outside business hours
  - Unusual access patterns
  - Multiple failed attempts
  - Credential exported (shouldn't happen)

Compliance and Reporting

Credential Inventory

# Generate credential inventory report
awx credentials list --all \
  --format json > credentials_inventory.json

# Analysis:
# - Total credentials
# - By type
# - By organization
# - Last used date
# - Rotation status

Access Audit

# Who can use each credential?
awx credentials get <ID> --related=access_list

# Credential usage history
awx activity_stream list \
  --object_type credential \
  --object_id <ID>

Compliance Reporting

Monthly Report:
  - Credentials requiring rotation
  - Unused credentials (> 90 days)
  - Credentials shared across teams (flag)
  - Failed access attempts
  - Vault sync status
  - External vault audit logs

Quarterly:
  - Full credential review
  - Access permission audit
  - Rotation policy compliance
  - Security incident review

Key Takeaways

✅ Custom credential types for any authentication need ✅ External vaults (HashiCorp/CyberArk) for dynamic secrets ✅ Credential rotation should be automated ✅ Least privilege principle for all credentials ✅ Segregation between environments ✅ Audit trail for compliance and security

What's Next

The next article covers Automation Mesh and Execution Environments - scaling AAP across regions, containerized execution, and building custom execution environments.

---

Next Article: Automation Mesh and Execution Environments →

---

Part of the Ansible Automation Platform 101 Series