Secrets Management and Credential Security

The Day Our Database Password Went Public on GitHub

2:15 PM, Friday afternoon. I was about to head out early when a message from GitHub's secret scanning bot appeared: "AWS credentials detected in public repository."

My heart sank. I checked the commit. Someone had pushed a .env file containing:

• AWS Access Keys (with admin permissions)

• Database passwords

• API keys for payment gateway

• JWT signing secrets

• Third-party service credentials

The commit had been public for 47 minutes. GitHub had already detected it and alerted us. But we had no idea who else had seen it.

Within 15 minutes, we detected unauthorized API calls from IP addresses in three countries. Someone had found the keys and was already probing our infrastructure. We spent the entire weekend in incident response:

• Rotating every credential in our entire infrastructure

• Investigating unauthorized access

• Notifying customers

• Filing security incident reports

The root cause? A developer running git add . without checking what was being committed. One careless commit cost us a weekend of work and nearly breached our entire production environment.

This article covers the zero-trust secrets management architecture I built to make sure this never happens again.

What You'll Learn

• Secrets management fundamentals and zero-trust principles

• HashiCorp Vault integration and best practices

• Cloud provider secret managers (AWS, Azure, GCP)

• Dynamic secrets and automatic rotation

• Secrets in CI/CD pipelines

• Git secret scanning and prevention

• Certificate and credential lifecycle management

The Secrets Problem

Modern applications have secrets everywhere:

Common anti-patterns:

• Hardcoded in source code

• Stored in config files in Git

• Baked into container images

• Shared via Slack/email

• Never rotated

• Overly broad permissions

HashiCorp Vault: The Foundation

Vault is the industry standard for secrets management.

Vault Installation and Setup

# docker-compose.yml
version: '3.8'
services:
  vault:
    image: hashicorp/vault:1.15
    container_name: vault
    ports:
      - "8200:8200"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: 'dev-only-token'
      VAULT_DEV_LISTEN_ADDRESS: '0.0.0.0:8200'
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data
      - vault-logs:/vault/logs
      - ./vault/config:/vault/config
    command: server

volumes:
  vault-data:
  vault-logs:

# vault/config/vault.hcl
storage "raft" {
  path    = "/vault/data"
  node_id = "node1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/vault/config/tls/vault.crt"
  tls_key_file  = "/vault/config/tls/vault.key"
}

api_addr = "https://vault.example.com:8200"
cluster_addr = "https://vault.example.com:8201"
ui = true

# Enable audit logging
audit {
  file {
    path = "/vault/logs/audit.log"
  }
}

Vault Initialization

#!/bin/bash
# scripts/init-vault.sh

# Initialize Vault
vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  > vault-keys.txt

# CRITICAL: Store these keys securely!
# Split among different trusted individuals
# Never store all keys in one place

# Unseal Vault (needs 3 of 5 keys)
vault operator unseal $(grep 'Key 1:' vault-keys.txt | awk '{print $NF}')
vault operator unseal $(grep 'Key 2:' vault-keys.txt | awk '{print $NF}')
vault operator unseal $(grep 'Key 3:' vault-keys.txt | awk '{print $NF}')

# Login with root token
ROOT_TOKEN=$(grep 'Initial Root Token:' vault-keys.txt | awk '{print $NF}')
vault login $ROOT_TOKEN

# Enable audit logging
vault audit enable file file_path=/vault/logs/audit.log

# Enable KV secrets engine
vault secrets enable -path=secret kv-v2

# Enable database secrets engine
vault secrets enable database

# Enable AWS secrets engine
vault secrets enable aws

Storing Secrets in Vault

# Store database credentials
vault kv put secret/database/prod \
  username=dbadmin \
  password=SuperSecure123! \
  host=db.example.com \
  port=5432

# Store API keys
vault kv put secret/api/stripe \
  publishable_key=pk_live_... \
  secret_key=sk_live_...

# Store TLS certificates
vault kv put secret/tls/api \
  certificate=@/path/to/cert.pem \
  private_key=@/path/to/key.pem

# Read secrets
vault kv get secret/database/prod
vault kv get -field=password secret/database/prod

Dynamic Secrets

The killer feature of Vault: secrets that are generated on-demand and automatically expire.

PostgreSQL Dynamic Secrets

# Configure PostgreSQL connection
vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly,readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb?sslmode=require" \
  username="vault" \
  password="vault-password"

# Create readonly role
vault write database/roles/readonly \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Create readwrite role
vault write database/roles/readwrite \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Generate dynamic credentials
vault read database/creds/readonly
# Returns:
# Key                Value
# ---                -----
# lease_id           database/creds/readonly/2f6a614c...
# lease_duration     1h
# lease_renewable    true
# password           A1a-y8sN8...
# username           v-root-readonly-x8fy8n...

AWS Dynamic Credentials

# Configure AWS secrets engine
vault write aws/config/root \
  access_key=$AWS_ACCESS_KEY_ID \
  secret_key=$AWS_SECRET_ACCESS_KEY \
  region=us-east-1

# Create role for S3 readonly access
vault write aws/roles/s3-readonly \
  credential_type=iam_user \
  policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": "*"
    }
  ]
}
EOF

# Generate AWS credentials
vault read aws/creds/s3-readonly

Application Integration

Node.js with Vault

// src/config/vault.js
const vault = require('node-vault');

class VaultClient {
  constructor() {
    this.client = vault({
      apiVersion: 'v1',
      endpoint: process.env.VAULT_ADDR || 'https://vault.example.com:8200',
      token: process.env.VAULT_TOKEN
    });
    
    this.secretCache = new Map();
    this.leases = new Map();
  }

  async getSecret(path) {
    // Check cache first
    if (this.secretCache.has(path)) {
      const cached = this.secretCache.get(path);
      if (cached.expiresAt > Date.now()) {
        return cached.data;
      }
    }

    // Read from Vault
    const result = await this.client.read(path);
    
    // Cache with TTL
    this.secretCache.set(path, {
      data: result.data.data,
      expiresAt: Date.now() + (3600 * 1000) // 1 hour
    });
    
    return result.data.data;
  }

  async getDynamicSecret(path) {
    const result = await this.client.read(path);
    
    // Store lease for renewal
    if (result.lease_id) {
      this.leases.set(path, {
        lease_id: result.lease_id,
        lease_duration: result.lease_duration
      });
      
      // Schedule renewal
      this.scheduleRenewal(path);
    }
    
    return result.data;
  }

  async scheduleRenewal(path) {
    const lease = this.leases.get(path);
    if (!lease) return;
    
    // Renew at 80% of lease duration
    const renewAt = lease.lease_duration * 0.8 * 1000;
    
    setTimeout(async () => {
      try {
        await this.client.renew({ lease_id: lease.lease_id });
        this.scheduleRenewal(path); // Schedule next renewal
      } catch (error) {
        console.error('Failed to renew lease:', error);
        // Fetch new credentials
        await this.getDynamicSecret(path);
      }
    }, renewAt);
  }

  async revokeLease(path) {
    const lease = this.leases.get(path);
    if (lease) {
      await this.client.revoke({ lease_id: lease.lease_id });
      this.leases.delete(path);
    }
  }
}

module.exports = new VaultClient();

// src/database.js
const { Pool } = require('pg');
const vault = require('./config/vault');

class DatabaseManager {
  constructor() {
    this.pool = null;
    this.credentialsPath = 'database/creds/readwrite';
  }

  async connect() {
    // Get dynamic credentials from Vault
    const creds = await vault.getDynamicSecret(this.credentialsPath);
    
    const dbConfig = await vault.getSecret('secret/database/prod');
    
    this.pool = new Pool({
      host: dbConfig.host,
      port: dbConfig.port,
      database: dbConfig.database,
      user: creds.username,
      password: creds.password,
      ssl: { rejectUnauthorized: false }
    });
    
    return this.pool;
  }

  async disconnect() {
    if (this.pool) {
      await this.pool.end();
      await vault.revokeLease(this.credentialsPath);
    }
  }
}

module.exports = new DatabaseManager();

Kubernetes Integration

External Secrets Operator

# external-secrets-operator.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: kube-system

---
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: production
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "production-role"
          serviceAccountRef:
            name: external-secrets

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  
  target:
    name: db-secret
    creationPolicy: Owner
  
  data:
    - secretKey: username
      remoteRef:
        key: database/prod
        property: username
    
    - secretKey: password
      remoteRef:
        key: database/prod
        property: password

Vault Agent Sidecar

# deployment-with-vault-agent.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "payment-service"
        vault.hashicorp.com/agent-inject-secret-database: "database/creds/readwrite"
        vault.hashicorp.com/agent-inject-template-database: |
          {{- with secret "database/creds/readwrite" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          {{- end }}
    spec:
      serviceAccountName: payment-service
      containers:
        - name: payment-service
          image: myapp:latest
          command: ["/bin/sh", "-c"]
          args:
            - source /vault/secrets/database && npm start

CI/CD Secrets Management

GitLab CI with Vault

# .gitlab-ci.yml
variables:
  VAULT_ADDR: "https://vault.example.com:8200"

.vault_auth:
  before_script:
    - apk add --no-cache curl jq
    - |
      # Get Vault token using JWT auth
      export VAULT_TOKEN=$(curl -s \
        --request POST \
        --data "{\"role\": \"gitlab-ci\", \"jwt\": \"$CI_JOB_JWT\"}" \
        $VAULT_ADDR/v1/auth/jwt/login | jq -r '.auth.client_token')

deploy:
  extends: .vault_auth
  stage: deploy
  script:
    # Get database credentials from Vault
    - |
      DB_CREDS=$(curl -s \
        -H "X-Vault-Token: $VAULT_TOKEN" \
        $VAULT_ADDR/v1/database/creds/readwrite)
      
      export DB_USERNAME=$(echo $DB_CREDS | jq -r '.data.username')
      export DB_PASSWORD=$(echo $DB_CREDS | jq -r '.data.password')
    
    # Run database migrations
    - npm run db:migrate
    
    # Deploy application
    - kubectl set image deployment/myapp myapp=myapp:${CI_COMMIT_SHA}

Secret Rotation

Automate credential rotation to limit blast radius.

// scripts/rotate-secrets.js
const vault = require('node-vault')({ endpoint: process.env.VAULT_ADDR });
const AWS = require('aws-sdk');

class SecretRotator {
  async rotateAWSKeys() {
    console.log('🔄 Rotating AWS keys...');
    
    // Get current AWS credentials from Vault
    const current = await vault.read('secret/aws/main');
    
    const iam = new AWS.IAM({
      accessKeyId: current.data.data.access_key,
      secretAccessKey: current.data.data.secret_key
    });
    
    // Create new access key
    const newKey = await iam.createAccessKey({
      UserName: 'vault-user'
    }).promise();
    
    // Update Vault with new credentials
    await vault.write('secret/aws/main', {
      data: {
        access_key: newKey.AccessKey.AccessKeyId,
        secret_key: newKey.AccessKey.SecretAccessKey,
        rotated_at: new Date().toISOString()
      }
    });
    
    // Wait for propagation
    await this.sleep(60000); // 1 minute
    
    // Delete old access key
    await iam.deleteAccessKey({
      UserName: 'vault-user',
      AccessKeyId: current.data.data.access_key
    }).promise();
    
    console.log('✅ AWS keys rotated successfully');
  }

  async rotateDatabase Credentials() {
    console.log('🔄 Rotating database credentials...');
    
    // Vault handles this automatically with dynamic secrets
    // But we can force rotation for static credentials
    
    const newPassword = this.generatePassword();
    
    // Update database
    await this.executeSQL(`ALTER USER admin PASSWORD '${newPassword}'`);
    
    // Update Vault
    await vault.write('secret/database/prod', {
      data: {
        password: newPassword,
        rotated_at: new Date().toISOString()
      }
    });
    
    console.log('✅ Database credentials rotated successfully');
  }

  generatePassword() {
    const length = 32;
    const charset = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%^&*';
    let password = '';
    for (let i = 0; i < length; i++) {
      password += charset.charAt(Math.floor(Math.random() * charset.length));
    }
    return password;
  }

  sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

// Run rotation
const rotator = new SecretRotator();
rotator.rotateAWSKeys()
  .then(() => rotator.rotateDatabaseCredentials())
  .catch(console.error);

Git Secret Scanning

Prevent secrets from ever reaching Git.

Pre-commit Hook with Gitleaks

#!/bin/bash
# .git/hooks/pre-commit

echo "🔍 Scanning for secrets..."

# Run gitleaks
gitleaks protect --staged --verbose

if [ $? -ne 0 ]; then
  echo "❌ Secrets detected! Commit blocked."
  echo "Remove secrets and try again."
  exit 1
fi

echo "✅ No secrets detected"
exit 0

GitLab Secret Detection

# .gitlab-ci.yml
secret_detection:
  stage: security
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:latest
  script:
    - /analyzer run
  artifacts:
    reports:
      secret_detection: gl-secret-detection-report.json
  allow_failure: false

GitHub Secret Scanning

GitHub automatically scans for leaked secrets, but add custom patterns:

# .github/secret_scanning.yml
patterns:
  - name: "Internal API Key"
    pattern: 'int_api_[a-zA-Z0-9]{32}'
  
  - name: "Database Connection String"
    pattern: 'postgresql://[^:]+:[^@]+@[^:]+:\d+/\w+'
  
  - name: "Private Key"
    pattern: '-----BEGIN (RSA|OPENSSH|DSA|EC|PGP) PRIVATE KEY-----'

Best Practices

1. Never Commit Secrets

# .gitignore
.env
.env.*
secrets/
*.pem
*.key
*_rsa
credentials.json

2. Use Short-Lived Credentials

# Vault: 1 hour default, 24 hour max
default_ttl="1h"
max_ttl="24h"

3. Principle of Least Privilege

# Vault policy: only what's needed
path "secret/data/myapp/*" {
  capabilities = ["read"]
}

4. Audit Everything

# Enable audit logging
vault audit enable file file_path=/vault/logs/audit.log

5. Rotate Regularly

# Automated rotation schedule
0 2 * * 0 /scripts/rotate-secrets.sh  # Weekly, Sunday 2 AM

Key Takeaways

✅ Never commit secrets to Git - Use secret managers ✅ Use dynamic secrets - Auto-generated, auto-expired ✅ Rotate credentials regularly - Limit blast radius ✅ Audit secret access - Know who accessed what ✅ Least privilege always - Grant minimum necessary access ✅ Scan for leaked secrets - Pre-commit hooks and CI/CD

What's Next

Now that secrets are managed, let's enforce security policies across the entire infrastructure. Next: Security Policy as Code with Open Policy Agent.

---

Next Article: Security Policy as Code →

---

Part of the DevSecOps 101 Series