Part 5: Building Observability Dashboards with CloudWatch

From Queries to Dashboards

Running adhoc queries solved immediate problems, but I needed persistent visibility into my systems. Building CloudWatch dashboards transformed how my team monitors production. This part covers everything I learned about creating effective observability dashboards.

CloudWatch Dashboards Overview

CloudWatch Dashboards provide real-time visualization of metrics and logs.

Dashboard Capabilities

• Widgets: Multiple visualization types

• Auto-refresh: Real-time updates

• Time range: Flexible time windows

• Sharing: Cross-account and public dashboards

• Layouts: Customizable grid layout

When I Use Dashboards vs Queries

Dashboards for:

• Continuous monitoring

• Team visibility

• SLA/SLO tracking

• High-level health checks

Queries for:

• Incident investigation

• Deep-dive analysis

• Adhoc exploration

• Root cause analysis

Creating Your First Dashboard

Via AWS Console

1. Navigate to CloudWatch → Dashboards

2. Click "Create dashboard"

3. Name your dashboard

4. Add widgets

Via AWS CLI

aws cloudwatch put-dashboard \
  --dashboard-name my-application-dashboard \
  --dashboard-body file://dashboard-definition.json

Via Terraform

resource "aws_cloudwatch_dashboard" "main" {
  dashboard_name = "my-application-dashboard"
  
  dashboard_body = jsonencode({
    widgets = [
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/Lambda", "Invocations", { stat = "Sum" }]
          ]
          period = 300
          stat   = "Average"
          region = "us-east-1"
          title  = "Lambda Invocations"
        }
      }
    ]
  })
}

Widget Types

1. Number Widget

Displays single metric value - perfect for KPIs.

{
  "type": "metric",
  "properties": {
    "metrics": [
      ["AWS/Lambda", "Errors", { "stat": "Sum", "label": "Total Errors" }]
    ],
    "view": "singleValue",
    "region": "us-east-1",
    "title": "Lambda Errors (Last Hour)",
    "period": 3600
  }
}

I use number widgets for:

• Error counts

• Active users

• Request counts

• Success rates

2. Line Graph Widget

Shows metrics over time - most common visualization.

{
  "type": "metric",
  "properties": {
    "metrics": [
      ["AWS/Lambda", "Duration", { "stat": "Average" }],
      [".", ".", { "stat": "p95" }],
      [".", ".", { "stat": "p99" }]
    ],
    "view": "timeSeries",
    "region": "us-east-1",
    "title": "Lambda Duration",
    "period": 300,
    "yAxis": {
      "left": {
        "label": "Milliseconds"
      }
    }
  }
}

I use line graphs for:

• Response times over time

• Request rates

• Error rates

• Resource utilization

3. Stacked Area Chart

Shows contribution of components to a total.

{
  "type": "metric",
  "properties": {
    "metrics": [
      ["AWS/ApiGateway", "Count", { "stat": "Sum" }, { "label": "2xx" }],
      ["...", { "stat": "Sum" }, { "label": "4xx" }],
      ["...", { "stat": "Sum" }, { "label": "5xx" }]
    ],
    "view": "timeSeries",
    "stacked": true,
    "region": "us-east-1",
    "title": "API Gateway Responses"
  }
}

4. Log Insights Widget

Displays results from CloudWatch Logs Insights queries.

{
  "type": "log",
  "properties": {
    "query": "SOURCE '/aws/lambda/my-function' | fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 10",
    "region": "us-east-1",
    "title": "Recent Errors",
    "stacked": false
  }
}

5. Pie Chart Widget

Shows distribution of categorical data.

{
  "type": "log",
  "properties": {
    "query": "SOURCE '/aws/lambda/my-function' | stats count() by @type",
    "region": "us-east-1",
    "title": "Lambda Event Types",
    "view": "pie"
  }
}

6. Bar Chart Widget

Compares values across categories.

{
  "type": "log",
  "properties": {
    "query": "SOURCE '/aws/apigateway/my-api' | stats count() by endpoint | sort count desc | limit 10",
    "region": "us-east-1",
    "title": "Top 10 Endpoints",
    "view": "bar"
  }
}

Dashboard Design Principles

Layout Strategy

I organize dashboards using this hierarchy:

Top Row: Golden Signals

• Request rate

• Error rate

• Latency (p50, p95, p99)

• Saturation

Middle Rows: Service Health

• Individual service metrics

• Dependencies

• Resource utilization

Bottom Rows: Detailed Logs

• Recent errors

• Slow requests

• Anomalies

Example Layout (Grid Coordinates)

[Request Rate] [Error Rate]    [P95 Latency]   [P99 Latency]    (Row 1)
[Lambda Invocations]            [API Gateway Requests]            (Row 2)
[DynamoDB Capacity]             [RDS Connections]                 (Row 3)
[Recent Errors Log]                                               (Row 4)
[Slow Queries Log]                                                (Row 5)

Color Coding Strategy

• Green: Healthy, normal operation

• Yellow: Warning, approaching threshold

• Red: Critical, requires attention

• Blue: Informational metrics

Real-World Dashboard Examples

Dashboard 1: Application Health Overview

This is my go-to dashboard for application health.

{
  "widgets": [
    {
      "type": "metric",
      "width": 6,
      "height": 3,
      "properties": {
        "metrics": [
          ["AWS/ApiGateway", "Count", { "stat": "Sum" }]
        ],
        "view": "singleValue",
        "title": "Total Requests (1h)",
        "period": 3600
      }
    },
    {
      "type": "metric",
      "width": 6,
      "height": 3,
      "properties": {
        "metrics": [
          ["AWS/ApiGateway", "5XXError", { "stat": "Sum" }]
        ],
        "view": "singleValue",
        "title": "5XX Errors (1h)",
        "period": 3600
      }
    },
    {
      "type": "metric",
      "width": 12,
      "height": 6,
      "properties": {
        "metrics": [
          ["AWS/Lambda", "Duration", { "stat": "p50" }],
          ["...", { "stat": "p95" }],
          ["...", { "stat": "p99" }]
        ],
        "view": "timeSeries",
        "title": "Response Time Percentiles",
        "yAxis": {
          "left": {
            "label": "ms"
          }
        }
      }
    },
    {
      "type": "log",
      "width": 24,
      "height": 6,
      "properties": {
        "query": "SOURCE '/aws/lambda/my-function' | fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20",
        "region": "us-east-1",
        "title": "Recent Errors"
      }
    }
  ]
}

Dashboard 2: API Performance Dashboard

Focused on API health and performance:

Widgets:

1. Request Rate Over Time

stats count() as requests by bin(1m)

1. Error Rate Percentage

stats 
  count() as total,
  count(if(statusCode >= 400, 1, 0)) as errors
  by bin(5m)
| fields bin(5m), errors * 100.0 / total as error_pct

1. Latency by Endpoint

fields endpoint, duration
| stats 
    pct(duration, 50) as p50,
    pct(duration, 95) as p95,
    pct(duration, 99) as p99
    by endpoint

1. Top Error Messages

filter statusCode >= 400
| parse @message /error[=:]?\s*(?<error>[^,\n]+)/
| stats count() by error
| sort count desc
| limit 10

1. Slowest Endpoints

fields endpoint, duration
| stats avg(duration) as avg_ms by endpoint
| sort avg_ms desc
| limit 10

Dashboard 3: Infrastructure Health

Monitoring underlying AWS services:

{
  "widgets": [
    {
      "type": "metric",
      "properties": {
        "metrics": [
          ["AWS/RDS", "CPUUtilization"],
          [".", "DatabaseConnections"],
          [".", "FreeableMemory"]
        ],
        "title": "RDS Metrics"
      }
    },
    {
      "type": "metric",
      "properties": {
        "metrics": [
          ["AWS/Lambda", "ConcurrentExecutions"],
          [".", "Throttles"]
        ],
        "title": "Lambda Concurrency"
      }
    },
    {
      "type": "metric",
      "properties": {
        "metrics": [
          ["AWS/DynamoDB", "ConsumedReadCapacityUnits"],
          [".", "ConsumedWriteCapacityUnits"]
        ],
        "title": "DynamoDB Capacity"
      }
    }
  ]
}

Dashboard 4: Customer Experience Dashboard

Focused on user-facing metrics:

Key Metrics:

1. Availability (Success Rate)

stats 
  count() as total,
  count(if(statusCode < 400, 1, 0)) as successful
| fields successful * 100.0 / total as availability_pct

1. Apdex Score

fields duration,
  if(duration < 100, "satisfied",
    if(duration < 400, "tolerating", "frustrated")
  ) as category
| stats count() by category

1. Active Users

fields userId
| filter ispresent(userId)
| stats count_distinct(userId) by bin(5m)

1. User Errors

fields userId, errorMessage
| filter ispresent(errorMessage)
| stats count() as error_count by userId
| sort error_count desc
| limit 10

Log-Based Metrics

Transform log queries into CloudWatch metrics for alerting.

Creating Log Metric Filters

1. Go to Log Group → Metric Filters

2. Create filter pattern

3. Assign metric name and namespace

4. Use in dashboards and alarms

Example: Error Rate Metric

Filter pattern:

[timestamp, request_id, level = ERROR*, ...]

Metric:

• Namespace: MyApp/Errors

• Metric name: ErrorCount

• Value: 1

Query Log-Based Metrics

SELECT SUM(ErrorCount) 
FROM "MyApp/Errors"

Real Example: Business Metrics from Logs

Track Order Completions:

Filter pattern:

{ $.event_type = "order_completed" }

Metric:

• Namespace: MyApp/Business

• Metric name: OrdersCompleted

• Value: $.order_value

Dashboard query:

stats sum(OrdersCompleted) by bin(1h)

Advanced Dashboard Patterns

Pattern 1: RED Method Dashboard

Rate, Errors, Duration

# Rate
stats count() as request_rate by bin(1m)

# Errors
stats count(if(statusCode >= 500, 1, 0)) as error_count by bin(1m)

# Duration
stats 
  pct(duration, 50) as p50,
  pct(duration, 95) as p95,
  pct(duration, 99) as p99
  by bin(1m)

Pattern 2: USE Method Dashboard

Utilization, Saturation, Errors

# Utilization
SELECT AVG(CPUUtilization) FROM AWS/EC2

# Saturation
SELECT MAX(QueueLength) FROM AWS/SQS

# Errors
stats count(if(@message like /ERROR/, 1, 0)) by bin(5m)

Pattern 3: Golden Signals Dashboard

Latency, Traffic, Errors, Saturation

1. Latency Widget:

stats pct(duration, 95) by bin(5m)

1. Traffic Widget:

stats count() by bin(1m)

1. Errors Widget:

stats count(if(statusCode >= 400, 1, 0)) / count() * 100 as error_pct by bin(5m)

1. Saturation Widget:

# Lambda concurrency
SELECT MAX(ConcurrentExecutions) FROM AWS/Lambda

# Or custom metric
stats count_distinct(active_connections) by bin(1m)

Pattern 4: Drill-Down Dashboard

Create linked dashboards:

• High-level overview

• Service-specific details

• Component-level deep dive

Dashboard Variables

Use variables for flexibility:

Time Range Variable

All widgets respect dashboard time range selector.

Region Variable

{
  "widgets": [
    {
      "type": "metric",
      "properties": {
        "region": "${AWS::Region}",
        "metrics": [...]
      }
    }
  ]
}

Dynamic Dimensions

stats count() by ${dimension}

Alarm Integration

Connect alarms to dashboard widgets:

Widget with Alarm

{
  "type": "metric",
  "properties": {
    "metrics": [
      ["AWS/Lambda", "Errors", { "stat": "Sum" }]
    ],
    "annotations": {
      "horizontal": [
        {
          "label": "Error threshold",
          "value": 10,
          "fill": "above"
        }
      ]
    }
  }
}

Alarm Status Widget

{
  "type": "alarm",
  "properties": {
    "alarms": [
      "arn:aws:cloudwatch:us-east-1:123456789012:alarm:HighErrorRate"
    ],
    "title": "Critical Alarms"
  }
}

Dashboard Auto-Refresh

Configure refresh intervals:

• 10 seconds

• 1 minute (default)

• 2 minutes

• 5 minutes

• 15 minutes

For production monitoring, I use 1-minute refresh.

Sharing Dashboards

Within AWS Account

Dashboards are accessible to users with CloudWatch permissions.

Cross-Account Dashboards

aws cloudwatch put-dashboard \
  --dashboard-name shared-dashboard \
  --dashboard-body file://dashboard.json \
  --account-id TARGET_ACCOUNT_ID

Public Dashboards

Make dashboards publicly accessible (useful for status pages):

1. Go to dashboard settings

2. Enable "Public dashboard"

3. Get shareable URL

Dashboard Best Practices

From my experience building dozens of dashboards:

1. Start with Golden Signals

Every dashboard should show:

• Request rate

• Error rate

• Response time

• Resource saturation

2. Use Consistent Time Windows

• Real-time monitoring: Last 1 hour

• Trend analysis: Last 24 hours

• Capacity planning: Last 7 days

3. Group Related Metrics

Don't mix unrelated services in one row.

4. Include Context

Add text widgets with:

• Dashboard purpose

• Alert thresholds

• Runbook links

5. Optimize Query Performance

• Use specific time ranges

• Apply filters early

• Limit result sets

• Cache expensive queries

6. Test Dashboard Load Time

Large dashboards can be slow:

• Limit to 20-30 widgets

• Use metric math when possible

• Consider separate dashboards for details

7. Include Recent Logs

Always add a log widget showing recent errors:

filter @message like /ERROR|FATAL/
| sort @timestamp desc
| limit 20

Mobile Dashboard Access

CloudWatch mobile app provides dashboard access on-call:

1. Download CloudWatch app (iOS/Android)

2. Con to AWS account

3. View dashboards

4. Receive alarm notifications

Dashboard as Code

Manage dashboards in version control:

Dashboard JSON in Git

dashboards/
  ├── application-health.json
  ├── infrastructure.json
  └── customer-experience.json

Automated Deployment

#!/bin/bash
for dashboard in dashboards/*.json; do
  name=$(basename $dashboard .json)
  aws cloudwatch put-dashboard \
    --dashboard-name $name \
    --dashboard-body file://$dashboard
done

Terraform Module

module "dashboards" {
  source = "./modules/cloudwatch-dashboards"
  
  dashboards = {
    "app-health" = file("dashboards/app-health.json")
    "infra"      = file("dashboards/infra.json")
  }
}

Cost Optimization

Dashboard costs are minimal but consider:

• Log Insights queries: $0.005/GB scanned

• Custom metrics: $0.30/metric/month

• API requests: Usually within free tier

Tips:

1. Use longer time intervals for less-critical metrics

2. Aggregate data before visualization

3. Archive old dashboards

4. Use efficient queries

Key Takeaways

• CloudWatch dashboards provide real-time visibility

• Widget types: number, line graph, logs, pie, bar

• Follow golden signals: latency, traffic, errors, saturation

• Log Insights widgets bring queries into dashboards

• Use log-based metrics for business KPIs

• Organize dashboards by audience and use case

• Include alarms and runbook links

• Manage dashboards as code for version control

• Test dashboard load times with many widgets

• Mobile app enables on-call access

In Part 6, we'll explore CloudWatch query best practices, performance optimization, cost management, and query patterns that scale to production workloads.