Infrastructure as Code Security

The $50,000 Mistake: When a Public S3 Bucket Cost More Than Money

3:42 AM. AWS GuardDuty alert: "Unusual API activity detected." Then another. And another. By the time I woke up at 7 AM, I had 247 security alerts and a $50,000 surprise AWS bill.

A Terraform misconfiguration had left an S3 bucket publicly accessible. In it? Database backups. Customer data. API keys. Everything. For six hours, attackers had been exfiltrating data and spinning up EC2 instances for cryptomining—all billed to us.

The root cause? One line in a Terraform file:

acl = "public-read"  # ← This destroyed us

Someone copy-pasted example code from a blog post. It passed code review. It deployed to production. Nobody caught it until AWS sent the bill.

That incident cost us$50,000 in AWS charges, weeks of incident response, mandatory breach notifications to 12,000 customers, and regulatory fines. But the worst part? It was 100% preventable. Infrastructure as Code security scanning would have caught it before it ever reached Git.

This article covers the IaC security strategy I built from the ashes of that disaster.

What You'll Learn

• Infrastructure as Code security fundamentals

• Terraform security scanning with Checkov, tfsec, and Terrascan

• CloudFormation and ARM template security

• Policy as Code with Open Policy Agent

• IaC security in CI/CD pipelines

• Cloud security posture management

• Drift detection and remediation

The IaC Security Challenge

Infrastructure as Code has revolutionized how we manage cloud resources. But it's also code—and code has vulnerabilities.

IaC security concerns:

• Misconfigurations (public buckets, open security groups)

• Insufficient encryption

• Overly permissive IAM policies

• Missing logging and monitoring

• Non-compliant configurations

• Hardcoded secrets

• Resource exposure

Terraform Security with Checkov

Checkov is my preferred IaC scanner—comprehensive, actively maintained, and supports multiple IaC frameworks.

Basic Checkov Scanning

# Scan current directory
checkov -d .

# Scan specific file
checkov -f main.tf

# Output JSON for automation
checkov -d . -o json > checkov-results.json

# Only show failed checks
checkov -d . --compact --quiet

# Scan with specific framework
checkov -d . --framework terraform

# Check specific policies
checkov -d . --check CKV_AWS_20,CKV_AWS_21

GitLab CI Integration

# .gitlab-ci.yml
stages:
  - validate
  - security
  - deploy

terraform_validate:
  stage: validate
  image: hashicorp/terraform:1.6
  script:
    - terraform init -backend=false
    - terraform validate
    - terraform fmt -check -recursive

checkov_scan:
  stage: security
  image: bridgecrew/checkov:latest
  script:
    # Run Checkov scan
    - checkov -d . -o json -o junitxml -o cli
    
    # Check results
    - |
      FAILED=$(cat results_junitxml.xml | grep -o 'failures="[0-9]*"' | grep -o '[0-9]*')
      echo "Checkov found $FAILED policy violations"
      
      if [ "$FAILED" -gt 0 ]; then
        echo "❌ Infrastructure security violations detected!"
        cat results_cli.txt
        exit 1
      fi
  
  artifacts:
    reports:
      junit: results_junitxml.xml
    paths:
      - results_json.json
      - results_cli.txt
    when: always
    expire_in: 30 days
  
  allow_failure: false

Real Vulnerabilities Caught by Checkov

1. Public S3 Bucket (The One That Cost Us)

# ❌ BAD - Public access
resource "aws_s3_bucket" "data" {
  bucket = "customer-data-backup"
  acl    = "public-read"  # CRITICAL VULNERABILITY
}

resource "aws_s3_bucket_public_access_block" "data" {
  bucket = aws_s3_bucket.data.id
  
  block_public_acls       = false  # ❌
  block_public_policy     = false  # ❌
  ignore_public_acls      = false  # ❌
  restrict_public_buckets = false  # ❌
}

# Checkov catches this:
Check: CKV_AWS_21: "Ensure the S3 bucket has MFA Delete enabled"
Check: CKV_AWS_18: "Ensure the S3 bucket has access logging enabled"
Check: CKV_AWS_19: "Ensure the S3 bucket has server-side encryption enabled"
Check: CKV_AWS_20: "Ensure the S3 bucket does not allow READ permissions"
Check: CKV_AWS_54: "Ensure S3 bucket has block public ACLS enabled"
Check: CKV_AWS_55: "Ensure S3 bucket has block public policy enabled"
Check: CKV_AWS_56: "Ensure S3 bucket has ignore public ACLs enabled"
Check: CKV_AWS_53: "Ensure S3 bucket has restrict public buckets enabled"

Fixed version:

# ✅ GOOD - Secure S3 bucket
resource "aws_s3_bucket" "data" {
  bucket = "customer-data-backup"
  # No ACL - defaults to private
}

resource "aws_s3_bucket_public_access_block" "data" {
  bucket = aws_s3_bucket.data.id
  
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "data" {
  bucket = aws_s3_bucket.data.id
  
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.s3.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_versioning" "data" {
  bucket = aws_s3_bucket.data.id
  
  versioning_configuration {
    status     = "Enabled"
    mfa_delete = "Enabled"
  }
}

resource "aws_s3_bucket_logging" "data" {
  bucket = aws_s3_bucket.data.id
  
  target_bucket = aws_s3_bucket.logs.id
  target_prefix = "s3-access-logs/"
}

2. Overly Permissive Security Group

# ❌ BAD - Open to the world
resource "aws_security_group" "web" {
  name        = "web-server"
  description = "Web server security group"
  
  ingress {
    description = "SSH"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # ❌ CRITICAL
  }
  
  ingress {
    description = "RDP"
    from_port   = 3389
    to_port     = 3389
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # ❌ CRITICAL
  }
  
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Fixed version:

# ✅ GOOD - Restricted access
resource "aws_security_group" "web" {
  name        = "web-server"
  description = "Web server security group"
  
  # HTTPS from ALB only
  ingress {
    description     = "HTTPS from ALB"
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }
  
  # No direct SSH - use Session Manager
  # No RDP
  
  # Specific egress rules
  egress {
    description = "HTTPS to internet"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  
  egress {
    description     = "Database"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.db.id]
  }
}

3. Unencrypted RDS Instance

# ❌ BAD - No encryption
resource "aws_db_instance" "main" {
  identifier        = "customer-db"
  engine            = "postgres"
  instance_class    = "db.t3.medium"
  allocated_storage = 100
  
  username = "admin"
  password = "SuperSecret123"  # ❌ Hardcoded password!
  
  storage_encrypted = false  # ❌ No encryption
  
  backup_retention_period = 0  # ❌ No backups
  
  publicly_accessible = true  # ❌ PUBLIC DATABASE
  
  skip_final_snapshot = true  # ❌ No final snapshot
}

Fixed version:

# ✅ GOOD - Secure RDS
resource "aws_db_instance" "main" {
  identifier        = "customer-db"
  engine            = "postgres"
  engine_version    = "15.3"
  instance_class    = "db.t3.medium"
  allocated_storage = 100
  
  # Password from Secrets Manager
  username               = "admin"
  manage_master_user_password = true
  master_user_secret_kms_key_id = aws_kms_key.rds.id
  
  # Encryption
  storage_encrypted = true
  kms_key_id       = aws_kms_key.rds.arn
  
  # Backups
  backup_retention_period   = 30
  backup_window            = "03:00-04:00"
  maintenance_window       = "mon:04:00-mon:05:00"
  copy_tags_to_snapshot    = true
  delete_automated_backups = false
  deletion_protection      = true
  
  # Network
  publicly_accessible    = false
  db_subnet_group_name   = aws_db_subnet_group.private.name
  vpc_security_group_ids = [aws_security_group.db.id]
  
  # Logging
  enabled_cloudwatch_logs_exports = ["postgresql", "upgrade"]
  
  # Monitoring
  monitoring_interval = 60
  monitoring_role_arn = aws_iam_role.rds_monitoring.arn
  
  # Performance Insights
  performance_insights_enabled    = true
  performance_insights_kms_key_id = aws_kms_key.rds.arn
  
  skip_final_snapshot = false
  final_snapshot_identifier = "customer-db-final-snapshot"
  
  tags = {
    Environment = "production"
    Compliance  = "pci-dss"
  }
}

tfsec for Fast Terraform Scanning

tfsec is incredibly fast and great for pre-commit hooks.

# Basic scan
tfsec .

# JSON output
tfsec . --format json > tfsec-results.json

# Sarif for IDE integration
tfsec . --format sarif > tfsec-results.sarif

# Only critical/high
tfsec . --minimum-severity HIGH

# Exclude specific checks
tfsec . --exclude aws-s3-enable-versioning

# Custom check location
tfsec . --custom-check-dir ./custom-checks

Pre-commit Hook

#!/bin/bash
# .git/hooks/pre-commit

echo "🔍 Running tfsec security scan..."

# Find changed Terraform files
TF_FILES=$(git diff --cached --name-only --diff-filter=ACM | grep '\.tf$')

if [ -z "$TF_FILES" ]; then
  echo "✅ No Terraform files changed"
  exit 0
fi

# Run tfsec
tfsec . --minimum-severity HIGH --force-all-dirs

if [ $? -ne 0 ]; then
  echo "❌ tfsec found security issues"
  echo "Fix issues or use --no-verify to skip (not recommended)"
  exit 1
fi

echo "✅ tfsec scan passed"
exit 0

Terrascan for Policy-Based Scanning

Terrascan supports OPA-based custom policies.

# .gitlab-ci.yml
terrascan_scan:
  stage: security
  image: tenable/terrascan:latest
  script:
    - terrascan scan -t terraform -d .
    
    # Scan with specific policies
    - terrascan scan -t terraform -d . -p AWS -p AZURE
    
    # JSON output
    - terrascan scan -t terraform -d . -o json > terrascan-results.json
    
    # SARIF for GitHub integration
    - terrascan scan -t terraform -d . -o sarif > terrascan-results.sarif
  
  artifacts:
    paths:
      - terrascan-results.json
      - terrascan-results.sarif

Custom Terrascan Policies

# policies/aws_ec2_instance_type.rego
package accurics

# Deny expensive instance types in non-production
ec2ExpensioveInstanceTypes[instance.id] {
  instance := input.aws_instance[_]
  
  # Get environment tag
  environment := instance.config.tags.Environment
  environment != "production"
  
  # Expensive instance types
  expensive_types := ["c5.24xlarge", "r5.24xlarge", "m5.24xlarge"]
  instance.config.instance_type[_] == expensive_types[_]
}

CloudFormation Security

For AWS CloudFormation templates:

# .gitlab-ci.yml
cfn_nag_scan:
  stage: security
  image: stelligent/cfn_nag:latest
  script:
    - cfn_nag_scan --input-path cloudformation/
    
    # JSON output
    - cfn_nag_scan --input-path cloudformation/ --output-format json > cfn-nag-results.json
    
    # Fail on warnings
    - cfn_nag_scan --input-path cloudformation/ --fail-on-warnings
  
  artifacts:
    paths:
      - cfn-nag-results.json

CloudFormation Guard

# Install cfn-guard
cargo install cfn-guard

# Validate against rules
cfn-guard validate -d template.yaml -r rules/

# Example rule
cat > rules/s3-rules.guard <<EOF
# S3 buckets must be encrypted
aws_s3_bucket {
  server_side_encryption_configuration EXISTS
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm == "AES256" or sse_algorithm == "aws:kms"
      }
    }
  }
}

# S3 buckets must block public access
aws_s3_bucket_public_access_block {
  block_public_acls == true
  block_public_policy == true
  ignore_public_acls == true
  restrict_public_buckets == true
}
EOF

Policy as Code with OPA

Open Policy Agent provides flexible policy enforcement.

# policies/terraform.rego
package terraform.security

import future.keywords

# Deny public S3 buckets
deny[msg] {
  resource := input.resource.aws_s3_bucket[name]
  resource.acl == "public-read"
  
  msg := sprintf("S3 bucket '%s' has public-read ACL", [name])
}

# Deny unencrypted EBS volumes
deny[msg] {
  resource := input.resource.aws_ebs_volume[name]
  resource.encrypted == false
  
  msg := sprintf("EBS volume '%s' is not encrypted", [name])
}

# Deny overly permissive security groups
deny[msg] {
  sg := input.resource.aws_security_group[name]
  rule := sg.ingress[_]
  
  contains_all_ips(rule.cidr_blocks)
  sensitive_port(rule.from_port)
  
  msg := sprintf("Security group '%s' allows %d from 0.0.0.0/0", [name, rule.from_port])
}

contains_all_ips(cidrs) {
  cidr := cidrs[_]
  cidr == "0.0.0.0/0"
}

sensitive_port(port) {
  sensitive_ports := [22, 3389, 3306, 5432, 1433, 27017]
  port == sensitive_ports[_]
}

# Require specific tags
deny[msg] {
  resource := input.resource[type][name]
  not resource.tags
  
  msg := sprintf("%s '%s' missing required tags", [type, name])
}

required_tags := ["Environment", "Owner", "CostCenter"]

deny[msg] {
  resource := input.resource[type][name]
  tags := object.keys(resource.tags)
  
  missing := [tag | tag := required_tags[_]; not tag in tags]
  count(missing) > 0
  
  msg := sprintf("%s '%s' missing tags: %v", [type, name, missing])
}

OPA Integration

# .gitlab-ci.yml
opa_check:
  stage: security
  image: openpolicyagent/opa:latest
  script:
    # Convert Terraform to JSON
    - terraform show -json plan.out > plan.json
    
    # Run OPA evaluation
    - opa eval -d policies/ -i plan.json "data.terraform.security.deny" --format pretty
    
    # Check for violations
    - |
      VIOLATIONS=$(opa eval -d policies/ -i plan.json "data.terraform.security.deny" --format raw | jq '. | length')
      if [ "$VIOLATIONS" -gt 0 ]; then
        echo "❌ Found $VIOLATIONS policy violations"
        opa eval -d policies/ -i plan.json "data.terraform.security.deny"
        exit 1
      fi

Drift Detection

Detect when infrastructure diverges from IaC definitions.

AWS Config for Drift Detection

# terraform/config.tf
resource "aws_config_configuration_recorder" "main" {
  name     = "config-recorder"
  role_arn = aws_iam_role.config.arn
  
  recording_group {
    all_supported = true
    include_global_resource_types = true
  }
}

resource "aws_config_delivery_channel" "main" {
  name           = "config-delivery"
  s3_bucket_name = aws_s3_bucket.config.id
  
  snapshot_delivery_properties {
    delivery_frequency = "Six_Hours"
  }
  
  depends_on = [aws_config_configuration_recorder.main]
}

# Managed rules
resource "aws_config_config_rule" "s3_bucket_public_read_prohibited" {
  name = "s3-bucket-public-read-prohibited"
  
  source {
    owner             = "AWS"
    source_identifier = "S3_BUCKET_PUBLIC_READ_PROHIBITED"
  }
  
  depends_on = [aws_config_configuration_recorder.main]
}

resource "aws_config_config_rule" "encrypted_volumes" {
  name = "encrypted-volumes"
  
  source {
    owner             = "AWS"
    source_identifier = "ENCRYPTED_VOLUMES"
  }
  
  depends_on = [aws_config_configuration_recorder.main]
}

Terraform Drift Detection

#!/bin/bash
# scripts/detect-drift.sh

echo "🔍 Detecting infrastructure drift..."

# Refresh Terraform state
terraform refresh

# Check for drift
terraform plan -detailed-exitcode

EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ]; then
  echo "✅ No drift detected"
  exit 0
elif [ $EXIT_CODE -eq 2 ]; then
  echo "⚠️ Drift detected!"
  
  # Generate drift report
  terraform plan -no-color > drift-report.txt
  
  # Send alert
  curl -X POST $SLACK_WEBHOOK \
    -H 'Content-Type: application/json' \
    -d "{
      \"text\": \"🚨 Infrastructure drift detected!\",
      \"attachments\": [{
        \"color\": \"warning\",
        \"text\": \"See pipeline for details\"
      }]
    }"
  
  exit 1
else
  echo "❌ Error running terraform plan"
  exit 1
fi

Best Practices

1. Scan Before Commit

# Pre-commit hook
checkov -d . --quiet --compact
tfsec . --minimum-severity HIGH

2. Multiple Scanning Tools

Different tools catch different issues:

parallel_scans:
  parallel:
    matrix:
      - TOOL: [checkov, tfsec, terrascan]

3. Policy as Code

# Codify your security requirements
deny[msg] {
  # Your security policy here
}

4. Continuous Scanning

schedule:
  - cron: '0 */6 * * *'  # Every 6 hours

5. Automated Remediation

# Self-healing infrastructure
resource "aws_config_remediation_configuration" "s3_block_public" {
  config_rule_name = aws_config_config_rule.s3_bucket_public_read_prohibited.name
  
  target_type      = "SSM_DOCUMENT"
  target_identifier = "AWS-PublishSNSNotification"
  
  automatic = true
  maximum_automatic_attempts = 5
  retry_attempt_seconds      = 60
}

Key Takeaways

✅ Scan IaC before deployment - Catch misconfigurations early ✅ Use multiple tools - Checkov, tfsec, Terrascan complement each other ✅ Enforce policies - OPA for custom security requirements ✅ Detect drift - Monitor for manual changes ✅ Automate remediation - Fix common issues automatically ✅ Continuous monitoring - Security doesn't end at deployment

What's Next

Secure infrastructure needs secure secrets. In the next article, we'll cover secrets management—HashiCorp Vault integration, secret rotation, and building a zero-trust secrets architecture.

---

Next Article: Secrets Management and Credential Security →

---

Part of the DevSecOps 101 Series