Chef Server Management

My Journey to Centralized Management

Managing a handful of servers with chef-client in local mode was fine initially, but when my infrastructure grew to 50+ nodes across multiple environments, I needed Chef Server. The transition wasn't just about scale - it was about gaining centralized control, policy enforcement, and the ability to manage nodes consistently across teams.

Setting up and managing Chef Server taught me valuable lessons about architecture, high availability, node bootstrapping, and the operational patterns that make large-scale automation sustainable. This article shares those insights.

Understanding Chef Server Architecture

Chef Server is the central hub that stores and distributes your automation code and data.

Components

┌──────────────────────────────────────────┐
│          Chef Workstation                │
│    (Where you develop cookbooks)         │
└────────────┬─────────────────────────────┘
             │ knife upload/download
             ▼
┌──────────────────────────────────────────┐
│          Chef Infra Server               │
│  ┌────────────────────────────────────┐  │
│  │  PostgreSQL (data storage)         │  │
│  ├────────────────────────────────────┤  │
│  │  Elasticsearch (search index)      │  │
│  ├────────────────────────────────────┤  │
│  │  Nginx (API gateway)               │  │
│  ├────────────────────────────────────┤  │
│  │  Erchef (API server)               │  │
│  ├────────────────────────────────────┤  │
│  │  Bookshelf (cookbook storage)      │  │
│  └────────────────────────────────────┘  │
└────────────┬─────────────────────────────┘
             │ chef-client runs
             ▼
┌──────────────────────────────────────────┐
│          Chef Clients (Nodes)            │
│   node1, node2, node3, ... node100       │
└──────────────────────────────────────────┘

What Chef Server Stores:

• Cookbooks and recipes

• Node data and attributes

• Roles and environments

• Data bags (including encrypted secrets)

• Policies and policy groups

• Search indices for querying infrastructure

Installing Chef Server

Standalone Installation

My initial production setup:

# Download Chef Server package
wget https://packages.chef.io/files/stable/chef-server/15.7.0/ubuntu/20.04/chef-server-core_15.7.0-1_amd64.deb

# Install
sudo dpkg -i chef-server-core_15.7.0-1_amd64.deb

# Configure
sudo chef-server-ctl reconfigure

# Create admin user
sudo chef-server-ctl user-create admin \
  Admin User \
  [email protected] \
  'SecurePassword123!' \
  --filename /tmp/admin.pem

# Create organization
sudo chef-server-ctl org-create infrastructure \
  'Infrastructure Team' \
  --association_user admin \
  --filename /tmp/infrastructure-validator.pem

Files to secure:

• /tmp/admin.pem - Admin user private key

• /tmp/infrastructure-validator.pem - Organization validator key

Post-Installation Configuration

# Install Chef Manage (web UI)
sudo chef-server-ctl install chef-manage
sudo chef-server-ctl reconfigure
sudo chef-manage-ctl reconfigure

# Test
curl https://chef-server.company.internal

High Availability Setup

For production, I run Chef Server in HA configuration:

My HA Architecture

                    ┌──────────────┐
                    │ Load Balancer│
                    │   (HAProxy)  │
                    └──────┬───────┘
                           │
        ┌──────────────────┼──────────────────┐
        │                  │                  │
   ┌────▼────┐       ┌────▼────┐       ┌────▼────┐
   │Frontend1│       │Frontend2│       │Frontend3│
   │  nginx  │       │  nginx  │       │  nginx  │
   │  erchef │       │  erchef │       │  erchef │
   └────┬────┘       └────┬────┘       └────┬────┘
        │                  │                  │
        └──────────────────┼──────────────────┘
                           │
              ┌────────────┼────────────┐
              │            │            │
         ┌────▼────┐  ┌────▼────┐  ┌────▼────┐
         │Backend 1│  │Backend 2│  │Backend 3│
         │PostgreSQL│ │PostgreSQL│ │PostgreSQL│
         └─────────┘  └─────────┘  └─────────┘

Benefits:

• No single point of failure

• Horizontal scaling

• Zero-downtime upgrades

• Geographic distribution

Configuration:

# /etc/opscode/chef-server.rb
topology "tier"
server "chef-frontend1.internal",
  :ipaddress => "10.0.1.10",
  :role => "frontend"
  
server "chef-backend1.internal",
  :ipaddress => "10.0.2.10",
  :role => "backend",
  :bootstrap => true
  
# Additional servers...

Bootstrapping Nodes

Bootstrapping connects a node to Chef Server and runs the first chef-client.

Basic Bootstrap

# Bootstrap Linux node via SSH
knife bootstrap node1.company.internal \
  --ssh-user ubuntu \
  --sudo \
  --identity-file ~/.ssh/id_rsa \
  --node-name web-server-01 \
  --run-list 'recipe[base],recipe[webserver]'

What happens:

1. Installs Chef Client on the node

2. Generates client key

3. Creates node object on Chef Server

4. Runs chef-client with specified run list

Bootstrap with Environment

knife bootstrap node1.company.internal \
  --ssh-user ubuntu \
  --sudo \
  --identity-file ~/.ssh/id_rsa \
  --node-name db-server-01 \
  --environment production \
  --run-list 'recipe[base],recipe[postgresql]'

Bootstrap Windows Nodes

knife bootstrap windows winrm node1.windows.internal \
  --winrm-user Administrator \
  --winrm-password 'SecurePass123!' \
  --node-name app-server-01 \
  --run-list 'recipe[base],recipe[iis]'

My Bootstrap Script

For consistent bootstrapping, I use a wrapper script:

#!/bin/bash
# bootstrap-node.sh

NODE_NAME=$1
NODE_IP=$2
ENVIRONMENT=$3
ROLE=$4

if [ -z "$NODE_NAME" ] || [ -z "$NODE_IP" ] || [ -z "$ENVIRONMENT" ] || [ -z "$ROLE" ]; then
  echo "Usage: $0 <node-name> <node-ip> <environment> <role>"
  exit 1
fi

echo "Bootstrapping $NODE_NAME ($NODE_IP) in $ENVIRONMENT as $ROLE"

knife bootstrap $NODE_IP \
  --ssh-user ubuntu \
  --sudo \
  --identity-file ~/.ssh/${ENVIRONMENT}_key \
  --node-name $NODE_NAME \
  --environment $ENVIRONMENT \
  --run-list "role[$ROLE]" \
  --json-attributes '{"tags": ["'$ENVIRONMENT'", "'$ROLE'"]}' \
  --bootstrap-version 18.4.12 \
  --verbose

if [ $? -eq 0 ]; then
  echo "Successfully bootstrapped $NODE_NAME"
  
  # Tag node for organization
  knife tag create $NODE_NAME $ENVIRONMENT $ROLE
  
  # Verify
  knife node show $NODE_NAME
else
  echo "Bootstrap failed for $NODE_NAME"
  exit 1
fi

Usage:

./bootstrap-node.sh web-prod-01 10.0.1.50 production webserver
./bootstrap-node.sh db-prod-01 10.0.2.51 production database

Managing Nodes with Knife

Node Operations

# List all nodes
knife node list

# Show node details
knife node show web-server-01

# Show node run list
knife node run_list show web-server-01

# Add recipe to run list
knife node run_list add web-server-01 'recipe[monitoring]'

# Remove recipe from run list
knife node run_list remove web-server-01 'recipe[monitoring]'

# Set entire run list
knife node run_list set web-server-01 'recipe[base],role[webserver]'

# Delete node
knife node delete web-server-01 --yes

Bulk Operations

# Run chef-client on all web servers
knife ssh "role:webserver" "sudo chef-client" \
  --ssh-user ubuntu \
  --identity-file ~/.ssh/id_rsa

# Run on specific environment
knife ssh "chef_environment:production" "sudo chef-client" \
  --ssh-user ubuntu \
  --identity-file ~/.ssh/id_rsa

# Specific command on filtered nodes
knife ssh "tags:pci-scope AND role:database" "uptime" \
  --ssh-user ubuntu \
  --identity-file ~/.ssh/id_rsa

Environments

Environments provide isolation between different infrastructure stages.

Creating Environments

environments/production.rb:

name 'production'
description 'Production environment'

cookbook 'webserver', '= 2.1.0'  # Pin specific versions
cookbook 'database', '= 1.5.0'

default_attributes(
  'webserver' => {
    'worker_processes' => 8,
    'worker_connections' => 4096
  },
  'monitoring' => {
    'enabled' => true,
    'retention_days' => 90
  }
)

override_attributes(
  'security' => {
    'ssh_password_auth' => false
  }
)

environments/staging.rb:

name 'staging'
description 'Staging environment'

cookbook 'webserver', '~> 2.0'  # Allow patch updates
cookbook 'database', '~> 1.5'

default_attributes(
  'webserver' => {
    'worker_processes' => 2,
    'worker_connections' => 1024
  },
  'monitoring' => {
    'enabled' => true,
    'retention_days' => 30
  }
)

Managing Environments

# Upload environment
knife environment from file environments/production.rb

# List environments
knife environment list

# Show environment
knife environment show production

# Move node to environment
knife node environment set web-server-01 production

My workflow:

1. Test cookbook changes in development

2. Promote to staging with version constraints

3. After validation, pin exact versions in production

Roles

Roles define run lists and attributes for groups of nodes.

Creating Roles

roles/webserver.rb:

name 'webserver'
description 'Web server role'

run_list(
  'recipe[base]',
  'recipe[security-baseline]',
  'recipe[nginx]',
  'recipe[application]',
  'recipe[monitoring::node_exporter]'
)

default_attributes(
  'nginx' => {
    'worker_processes' => 'auto',
    'keepalive_timeout' => 65
  }
)

override_attributes(
  'security' => {
    'firewall_enabled' => true,
    'allowed_ports' => [22, 80, 443]
  }
)

roles/database.rb:

name 'database'
description 'Database server role'

run_list(
  'recipe[base]',
  'recipe[security-baseline]',
  'recipe[postgresql::server]',
  'recipe[postgresql::backup]',
  'recipe[monitoring::postgres_exporter]'
)

default_attributes(
  'postgresql' => {
    'version' => '12',
    'max_connections' => 200,
    'shared_buffers' => '4GB'
  }
)

Managing Roles

# Upload role
knife role from file roles/webserver.rb

# List roles
knife role list

# Show role
knife role show webserver

# Assign role to node
knife node run_list set web-server-01 'role[webserver]'

# Search nodes by role
knife search node "role:webserver"

When I use roles:

• Grouping common functionality (webserver, database, cache)

• Defining standard run lists

• Setting role-specific attributes

Policy-Based Management

Policyfiles are the modern alternative to roles and environments.

Creating a Policyfile

Policyfile.rb:

name 'webserver'

default_source :supermarket

run_list 'base::default', 'nginx::default', 'application::default'

cookbook 'base', path: './cookbooks/base'
cookbook 'nginx', '~> 10.0'
cookbook 'application', git: 'https://github.com/myorg/app-cookbook.git'

override['nginx']['worker_processes'] = 4
override['application']['environment'] = 'production'

Using Policyfiles

# Install cookbooks
chef install Policyfile.rb

# Generate Policyfile.lock.json
chef update Policyfile.rb

# Upload to Chef Server
chef push production Policyfile.rb

# Bootstrap with policy
knife bootstrap node1.company.internal \
  --policy-name webserver \
  --policy-group production

Why I prefer Policyfiles:

• Explicit dependency management

• Lock files ensure consistency

• Better testing with Test Kitchen

• Cleaner than role/environment combinations

Search and Queries

Chef's search allows dynamic discovery of infrastructure.

Basic Search

# Search all nodes
knife search node "*:*"

# Search by role
knife search node "role:webserver"

# Search by environment
knife search node "chef_environment:production"

# Search by attribute
knife search node "platform:ubuntu"

# Combined search
knife search node "role:webserver AND chef_environment:production"

Using Search in Recipes

recipes/configure.rb:

# Find all database servers
db_servers = search(:node, "role:database AND chef_environment:#{node.chef_environment}")

# Extract IPs for connection pooling
db_ips = db_servers.map { |n| n['ipaddress'] }

template '/etc/myapp/database.yml' do
  variables(
    db_hosts: db_ips
  )
end

Real scenario: My application servers automatically discover database servers - no hardcoded IPs.

Advanced Search Queries

# Find nodes with specific attribute
web_nodes = search(:node, 'webserver_port:8080')

# Find nodes with tag
production_nodes = search(:node, 'tags:production')

# Find nodes updated recently
recent_nodes = search(:node, "ohai_time:[#{1.hour.ago.to_i} TO *]")

# Search data bags
users = search(:users, 'groups:developers')

Node Lifecycle Management

Decommissioning Nodes

# 1. Stop chef-client on the node
ssh node1 'sudo systemctl stop chef-client'

# 2. Delete from Chef Server
knife node delete web-server-01 --yes
knife client delete web-server-01 --yes

# 3. Clean up any lingering data
knife search node "name:web-server-01"  # Verify deleted

My cleanup script:

#!/bin/bash
# decommission-node.sh

NODE_NAME=$1

echo "Decommissioning $NODE_NAME"

# Delete node
knife node delete $NODE_NAME --yes

# Delete client
knife client delete $NODE_NAME --yes

# Verify cleanup
if knife node show $NODE_NAME 2>/dev/null; then
  echo "ERROR: Node still exists!"
  exit 1
else
  echo "Successfully decommissioned $NODE_NAME"
fi

Node Re-registration

If a node loses its client key:

# On the node
sudo rm /etc/chef/client.pem

# Delete client on server
knife client delete web-server-01 --yes

# Re-run chef-client (will re-register)
sudo chef-client

Monitoring Chef Server

Health Checks

# Overall status
sudo chef-server-ctl status

# Service-specific status
sudo chef-server-ctl status nginx
sudo chef-server-ctl status postgresql

# Test connectivity
sudo chef-server-ctl test

Logs

# View all logs
sudo chef-server-ctl tail

# Specific service logs
sudo chef-server-ctl tail nginx
sudo chef-server-ctl tail postgresql
sudo chef-server-ctl tail erchef

# Follow logs in real-time
sudo chef-server-ctl tail -f

Performance Monitoring

# Top processes
sudo chef-server-ctl top

# Gather logs for support
sudo chef-server-ctl gather-logs

Metrics I monitor:

• Chef client run success rate

• Average run duration

• API response times

• PostgreSQL connection pool usage

• Search index latency

Backup and Recovery

Backing Up Chef Server

# Create backup
sudo chef-server-ctl backup

# Backup location
ls /var/opt/chef-backup/

# Backup to specific location
sudo chef-server-ctl backup --backup-dir /mnt/backups/chef

My backup script:

#!/bin/bash
# backup-chef-server.sh

BACKUP_DIR="/mnt/backups/chef"
RETENTION_DAYS=30
DATE=$(date +%Y%m%d)

echo "Starting Chef Server backup..."

sudo chef-server-ctl backup --backup-dir $BACKUP_DIR

# Compress backup
cd $BACKUP_DIR
tar -czf chef-backup-$DATE.tar.gz *-$DATE

# Upload to S3
aws s3 cp chef-backup-$DATE.tar.gz s3://company-backups/chef/

# Clean up old backups
find $BACKUP_DIR -name "*.tar.gz" -mtime +$RETENTION_DAYS -delete

echo "Backup completed successfully"

Cron job:

0 2 * * * /usr/local/bin/backup-chef-server.sh

Restoring Chef Server

# Stop services
sudo chef-server-ctl stop

# Restore
sudo chef-server-ctl restore /path/to/backup

# Reconfigure
sudo chef-server-ctl reconfigure

# Start services
sudo chef-server-ctl start

Best Practices

1. Node Organization

Use tags for flexible organization:

# Tag nodes
knife tag create web-prod-01 production webserver pci-scope aws us-west-2

# Search by tags
knife search node "tags:pci-scope AND tags:production"

2. Run List Management

Keep run lists maintainable:

# Good: Use roles
run_list 'role[base]', 'role[webserver]'

# Avoid: Long lists of individual recipes
run_list 'recipe[apt]', 'recipe[ntp]', 'recipe[ssh]', ...

3. Cookbook Version Constraints

Pin versions in production:

# environment/production.rb
cookbook 'nginx', '= 2.1.0'  # Exact version

# environment/staging.rb
cookbook 'nginx', '~> 2.0'  # Pessimistic constraint

4. Regular Maintenance

# Clean up unused nodes
knife status --run-list | grep -v 'hours ago'

# Verify cookbook dependencies
knife cookbook test my_cookbook

# Update knife.rb for security
echo "ssl_verify_mode :verify_peer" >> ~/.chef/knife.rb

What's Next?

You now understand: ✅ Chef Server architecture and installation ✅ Node bootstrapping and lifecycle management ✅ Environments and roles for organization ✅ Policy-based management with Policyfiles ✅ Search and dynamic infrastructure discovery ✅ Backup and recovery procedures

Continue to Chef InSpec Compliance to learn about compliance automation, or explore Best Practices for production deployment patterns.

Chef Server is the backbone of enterprise Chef automation. Master these patterns, and you'll be able to manage infrastructure at any scale with confidence.

---

Ready to implement compliance as code? Continue to Chef InSpec Compliance