Chef Best Practices

My Evolution to Production Excellence

Three years ago, I deployed a cookbook that accidentally restarted all production web servers simultaneously, causing a 15-minute outage. That incident taught me the difference between code that "works" and code that's production-ready. Since then, I've developed patterns and practices that have kept my Chef automation running smoothly through thousands of deployments.

This article shares the hard-won lessons that transformed my Chef practice from functional to truly reliable.

The Golden Rules

1. Test Everything, Always

Never deploy untested code to production.

My testing checklist:

✓ Syntax check (Cookstyle)
✓ Unit tests (ChefSpec)
✓ Integration tests (Test Kitchen)
✓ Multi-platform validation
✓ Staging environment deployment
✓ Production canary deployment

2. Make Everything Idempotent

Resources should be safe to run multiple times.

# Bad - runs every time
execute 'restart_app' do
  command 'systemctl restart myapp'
end

# Good - only when needed
service 'myapp' do
  action :nothing
  subscribes :restart, 'template[/etc/myapp/config.yml]', :delayed
end

3. Use Version Control for Everything

All Chef code belongs in Git.

chef-repo/
├── .git/
├── cookbooks/
├── environments/
├── roles/
└── data_bags/

Never make changes directly on Chef Server.

4. Pin Versions in Production

Environment-specific version constraints:

# environments/production.rb
cookbook 'nginx', '= 2.1.0'      # Exact version

# environments/staging.rb  
cookbook 'nginx', '~> 2.0'       # Allow patches

# environments/development.rb
cookbook 'nginx', '>= 2.0'       # Latest

Cookbook Development Best Practices

Naming Conventions

Consistent naming prevents confusion:

# Cookbook names
my_app          # Good: lowercase, underscores
MyApp           # Bad: CamelCase
my-app          # Bad: hyphens

# Recipe names
default.rb      # Main recipe
install.rb      # Clear purpose
configure.rb
service.rb

# Attribute files
default.rb      # Default attributes
production.rb   # Environment-specific

Cookbook Structure

Standard layout I use:

my_cookbook/
├── attributes/
│   ├── default.rb
│   └── production.rb
├── recipes/
│   ├── default.rb
│   ├── install.rb
│   ├── configure.rb
│   └── service.rb
├── templates/
│   └── default/
│       ├── config.erb
│       └── service.erb
├── files/
│   └── default/
│       └── static-file.txt
├── libraries/
│   └── helpers.rb
├── resources/
│   └── my_resource.rb
├── spec/
│   └── unit/
│       └── recipes/
├── test/
│   └── integration/
│       └── default/
├── metadata.rb
├── README.md
├── CHANGELOG.md
└── Berksfile

Documentation Standards

metadata.rb:

name 'my_cookbook'
maintainer 'Infrastructure Team'
maintainer_email '[email protected]'
license 'Apache-2.0'
description 'Manages application deployment'
version '2.1.0'

supports 'ubuntu', '>= 18.04'
supports 'centos', '>= 7.0'

depends 'nginx', '~> 10.0'
depends 'postgresql', '~> 9.0'

chef_version '>= 15.0'

README.md:

# My Cookbook

## Description
Configures and deploys MyApp application.

## Requirements
- Chef 15+
- Ubuntu 18.04+ or CentOS 7+

## Attributes
See attributes/default.rb

## Usage
```ruby
include_recipe 'my_cookbook::default'

Testing

cookstyle .
chef exec rspec
kitchen test

License

Apache 2.0

**CHANGELOG.md:**
```markdown
# Changelog

## 2.1.0 (2026-01-07)
### Added
- Support for Ubuntu 22.04
- Health check endpoint configuration

### Changed
- Updated nginx dependency to 10.x

### Fixed
- Service restart on config change

Attribute Organization

Clear, namespaced attributes:

# attributes/default.rb

# Package versions
default['my_app']['version'] = '2.1.0'
default['my_app']['nginx_version'] = '1.18.0'

# Paths
default['my_app']['install_dir'] = '/opt/myapp'
default['my_app']['config_dir'] = '/etc/myapp'
default['my_app']['log_dir'] = '/var/log/myapp'

# Service configuration
default['my_app']['service_name'] = 'myapp'
default['my_app']['port'] = 8080
default['my_app']['workers'] = 4

# Environment-specific overrides
case node.chef_environment
when 'production'
  default['my_app']['workers'] = 16
  default['my_app']['debug'] = false
when 'development'
  default['my_app']['workers'] = 2
  default['my_app']['debug'] = true
end

# Feature flags
default['my_app']['features']['new_ui'] = false
default['my_app']['features']['beta_api'] = false

Resource Best Practices

Always Use Guards

# Good - idempotent
execute 'bundle-install' do
  command 'bundle install'
  cwd '/opt/myapp'
  not_if 'bundle check', cwd: '/opt/myapp'
end

# Good - creates file marker
execute 'app-setup' do
  command '/opt/myapp/setup.sh'
  creates '/opt/myapp/.initialized'
end

# Good - conditional check
package 'development-tools' do
  action :install
  only_if { node['my_app']['dev_mode'] }
end

Proper Notification Patterns

# Use delayed notifications (safer)
template '/etc/nginx/nginx.conf' do
  source 'nginx.conf.erb'
  notifies :reload, 'service[nginx]', :delayed
end

# Test before restarting
execute 'test-nginx' do
  command 'nginx -t'
  action :nothing
end

template '/etc/nginx/nginx.conf' do
  source 'nginx.conf.erb'
  notifies :run, 'execute[test-nginx]', :immediate
  notifies :reload, 'service[nginx]', :delayed
end

service 'nginx' do
  action [:enable, :start]
  supports reload: true, status: true, restart: true
end

File and Directory Management

# Create directory hierarchy
directory '/opt/myapp/config' do
  owner 'myapp'
  group 'myapp'
  mode '0755'
  recursive true
end

# Template with proper permissions
template '/opt/myapp/config/secrets.yml' do
  source 'secrets.yml.erb'
  owner 'myapp'
  group 'myapp'
  mode '0600'  # Restrictive for secrets
  sensitive true  # Hides content in logs
end

# Remote file with checksum validation
remote_file '/tmp/app.tar.gz' do
  source 'https://releases.company.internal/app-2.1.0.tar.gz'
  checksum 'abc123...'
  owner 'root'
  group 'root'
  mode '0644'
end

Security Best Practices

Never Hardcode Secrets

# Bad - exposed in Git
template '/etc/myapp/config.yml' do
  variables(
    db_password: 'SuperSecret123'
  )
end

# Good - use encrypted data bags
db_credentials = Chef::EncryptedDataBagItem.load('credentials', 'database')

template '/etc/myapp/config.yml' do
  variables(
    db_password: db_credentials['password']
  )
  sensitive true
end

Secure File Permissions

# Secrets should be restrictive
file '/etc/myapp/api_key' do
  content api_key
  owner 'myapp'
  group 'myapp'
  mode '0400'  # Read-only for owner
  sensitive true
end

# SSH keys
file '/home/deploy/.ssh/id_rsa' do
  content private_key
  owner 'deploy'
  group 'deploy'
  mode '0600'
  sensitive true
end

Use Chef Vault

# Create vault item
knife vault create credentials database \
  '{"username":"dbuser","password":"SecurePass123!"}' \
  --search 'role:database' \
  --admins 'admin'

# In recipe
require 'chef-vault'

db_creds = ChefVault::Item.load('credentials', 'database')

template '/etc/myapp/database.yml' do
  variables(
    username: db_creds['username'],
    password: db_creds['password']
  )
  sensitive true
end

Deployment Strategies

Canary Deployments

Test changes on subset of nodes first:

# 1. Tag canary nodes
knife tag create web-prod-01 canary
knife tag create web-prod-02 canary

# 2. Upload new cookbook version
knife cookbook upload webserver

# 3. Deploy to canary nodes only
knife ssh "tags:canary AND role:webserver" "sudo chef-client"

# 4. Monitor for issues (wait 30 minutes)

# 5. If successful, deploy to remaining nodes
knife ssh "NOT tags:canary AND role:webserver" "sudo chef-client"

Blue-Green Deployments

Maintain two identical environments:

# Tag nodes with color
knife tag create web-prod-01 blue
knife tag create web-prod-02 blue
knife tag create web-prod-03 green
knife tag create web-prod-04 green

# Deploy to green (inactive)
knife ssh "tags:green" "sudo chef-client"

# Switch load balancer to green

# Deploy to blue (now inactive)
knife ssh "tags:blue" "sudo chef-client"

Rolling Deployments

Update nodes in batches:

#!/bin/bash
# rolling-deploy.sh

NODES=$(knife search node "role:webserver" -i)
BATCH_SIZE=3

for batch in $(echo "$NODES" | xargs -n $BATCH_SIZE); do
  echo "Deploying to batch: $batch"
  
  for node in $batch; do
    knife ssh "name:$node" "sudo chef-client" &
  done
  
  wait
  
  echo "Batch complete. Monitoring..."
  sleep 300  # Wait 5 minutes between batches
done

Error Handling and Recovery

Defensive Cookbook Patterns

# Check prerequisites
ruby_block 'verify-prerequisites' do
  block do
    raise 'Database connection required' unless node['myapp']['db_host']
    raise 'Invalid port number' unless node['myapp']['port'].between?(1, 65535)
  end
end

# Graceful degradation
template '/etc/myapp/config.yml' do
  source 'config.yml.erb'
  variables lazy {
    {
      db_host: node['myapp']['db_host'] || 'localhost',
      cache_enabled: node['myapp']['cache_enabled'] || false
    }
  }
end

# Retry failed operations
remote_file '/tmp/package.deb' do
  source 'https://releases.company.internal/package.deb'
  retries 3
  retry_delay 10
end

Health Checks

# Verify service is responding
execute 'health-check' do
  command 'curl -f http://localhost:8080/health || exit 1'
  retries 5
  retry_delay 10
  action :nothing
  subscribes :run, 'service[myapp]', :delayed
end

Performance Optimization

Minimize Chef Runs

# Use lazy evaluation for expensive operations
node.default['myapp']['db_servers'] = lazy {
  search(:node, 'role:database').map { |n| n['ipaddress'] }
}

# Cache search results
ruby_block 'cache-db-servers' do
  block do
    node.run_state['db_servers'] ||= search(:node, 'role:database')
  end
end

# Use cached data
db_servers = node.run_state['db_servers']

Reduce Cookbook Size

# Use .chefignore to exclude unnecessary files
# .chefignore
.git
.kitchen
.github
spec/
test/
*.md
*.txt
Vagrantfile

Attribute Precedence Awareness

Use the right level:

# Default - for cookbook defaults
default['myapp']['port'] = 8080

# Normal - for node-specific values (persisted)
node.normal['myapp']['last_deployed'] = Time.now

# Override - for environment/role enforcement
override['myapp']['env'] = node.chef_environment

Monitoring and Observability

Chef Client Run Reporting

# Report handlers
chef_gem 'chef-handler-datadog' do
  compile_time true
end

require 'chef/handler/datadog'

chef_handler 'Chef::Handler::Datadog' do
  source 'chef/handler/datadog'
  arguments api_key: node['datadog']['api_key']
  supports report: true, exception: true
  action :enable
end

Logging Best Practices

# Use Chef::Log for important messages
Chef::Log.info("Deploying application version #{node['myapp']['version']}")
Chef::Log.warn("SSL certificates expiring in 30 days")
Chef::Log.error("Failed to connect to database")

# Custom logging
ruby_block 'log-deployment' do
  block do
    File.open('/var/log/myapp/deployments.log', 'a') do |f|
      f.puts "#{Time.now} - Deployed version #{node['myapp']['version']}"
    end
  end
end

Maintenance and Technical Debt

Regular Cookbook Audits

Monthly checklist:

✓ Update cookbook dependencies
✓ Review and update test coverage
✓ Check for deprecated Chef features
✓ Update platform support versions
✓ Review and clean up unused attributes
✓ Update documentation

Deprecation Management

# Warn about deprecated attributes
if node['myapp']['old_attribute']
  Chef::Log.warn(
    'Attribute old_attribute is deprecated. Use new_attribute instead.'
  )
  node.normal['myapp']['new_attribute'] = node['myapp']['old_attribute']
end

# Support old and new patterns
port = node['myapp']['port'] || node['myapp']['listen_port'] || 8080

Team Collaboration

Code Review Standards

Pull request checklist:

## Cookbook Changes
- [ ] Tests added/updated
- [ ] Documentation updated
- [ ] Version bumped in metadata.rb
- [ ] CHANGELOG.md updated
- [ ] Cookstyle passes
- [ ] All tests pass

## Testing Evidence
- [ ] ChefSpec output
- [ ] Test Kitchen results
- [ ] Staging deployment successful

## Deployment Plan
- [ ] Rollback plan documented
- [ ] Monitoring plan defined
- [ ] Stakeholders notified

Knowledge Sharing

Document tribal knowledge:

# Cookbook Troubleshooting Guide

## Common Issues

### Service Won't Start
Check logs: `/var/log/myapp/error.log`
Verify config: `myapp --check-config`

### Database Connection Failures
Verify credentials in data bag
Check network connectivity
Review pg_hba.conf settings

### Performance Issues
Increase workers: node['myapp']['workers']
Check memory usage
Review slow query logs

What's Next?

You now understand: ✅ Production-ready cookbook development patterns ✅ Security and secret management best practices ✅ Deployment strategies for zero downtime ✅ Error handling and recovery techniques ✅ Performance optimization approaches ✅ Team collaboration and maintenance

These best practices are the difference between automation that works and automation that's reliable, secure, and maintainable at scale.

---

Continue exploring Chef 101 or revisit Chef 101 Overview for the full series