Part 4: Querying AWS Services with CloudWatch Logs

Navigating AWS Service Logs

Each AWS service logs differently. Over time, I've learned the quirks and patterns of each service's logs. This part provides production-ready queries for the AWS services I work with most.

AWS Lambda Logs

Lambda is perhaps the most common source of CloudWatch logs in modern AWS architectures.

Lambda Log Structure

Lambda creates log streams per function invocation and includes automatic logging:

START RequestId: abc-123 Version: $LATEST
2024-01-15T10:30:45.123Z abc-123 INFO User logged in
END RequestId: abc-123
REPORT RequestId: abc-123 Duration: 125.45 ms Billed Duration: 126 ms Memory Size: 512 MB Max Memory Used: 95 MB

Finding Errors in Lambda

fields @timestamp, @message
| filter @type = "REPORT" or @message like /ERROR|Error|error/
| filter @message not like /START|END/
| sort @timestamp desc
| limit 50

Lambda Cold Start Analysis

fields @timestamp, @message
| filter @type = "REPORT"
| parse @message /Duration: (?<duration>[0-9.]+)/
| parse @message /Init Duration: (?<init_duration>[0-9.]+)/
| filter ispresent(init_duration)
| stats 
    count() as cold_starts,
    avg(init_duration) as avg_init_ms,
    avg(duration) as avg_duration_ms,
    max(init_duration) as max_init_ms
| display cold_starts, avg_init_ms, avg_duration_ms, max_init_ms

Lambda Performance Metrics

fields @timestamp, @message
| filter @type = "REPORT"
| parse @message /Duration: (?<duration>[0-9.]+)/
| parse @message /Billed Duration: (?<billed>[0-9]+)/
| parse @message /Memory Size: (?<mem_size>[0-9]+)/
| parse @message /Max Memory Used: (?<mem_used>[0-9]+)/
| stats 
    count() as invocations,
    avg(duration) as avg_duration_ms,
    pct(duration, 95) as p95_duration_ms,
    pct(duration, 99) as p99_duration_ms,
    avg(mem_used / mem_size * 100) as avg_memory_pct,
    max(mem_used / mem_size * 100) as max_memory_pct,
    sum(billed) / 1000 as total_billed_seconds

Lambda Timeouts

fields @timestamp, @message, @requestId
| filter @message like /Task timed out/
| sort @timestamp desc

Real Example: Lambda Error Tracking

fields @timestamp, @message, @requestId
| filter @message like /ERROR|Error|Exception/
| filter @message not like /START|END|REPORT/
| parse @message /ERROR\s+(?<error_type>[^\s:]+):?\s*(?<error_message>.*)/
| stats count() as occurrences by error_type
| sort occurrences desc

API Gateway Logs

API Gateway logs require explicit enablement but provide invaluable request/response data.

Enable API Gateway Logging

First, create a CloudWatch log role ARN in IAM, then enable logs for your API stage.

API Gateway Log Format

{
  "requestId": "abc-123",
  "ip": "192.168.1.1",
  "requestTime": "15/Jan/2024:10:30:45 +0000",
  "httpMethod": "GET",
  "routeKey": "$default",
  "status": 200,
  "protocol": "HTTP/1.1",
  "responseLength": 1234
}

Query API Gateway Access Logs

fields @timestamp, requestId, ip, httpMethod, path, status, responseTime
| filter status >= 400
| sort @timestamp desc
| limit 50

API Gateway Error Rate

fields @timestamp, status
| stats 
    count() as total_requests,
    count(if(status >= 400 and status < 500, 1, 0)) as client_errors,
    count(if(status >= 500, 1, 0)) as server_errors
    by bin(5m) as interval
| fields interval,
    client_errors * 100.0 / total_requests as client_error_pct,
    server_errors * 100.0 / total_requests as server_error_pct
| sort interval desc

API Gateway Performance by Endpoint

fields @timestamp, httpMethod, path, responseTime, status
| filter ispresent(responseTime)
| stats 
    count() as requests,
    avg(responseTime) as avg_ms,
    pct(responseTime, 50) as p50_ms,
    pct(responseTime, 95) as p95_ms,
    pct(responseTime, 99) as p99_ms,
    max(responseTime) as max_ms
    by concat(httpMethod, " ", path) as endpoint
| filter requests > 100
| sort p95_ms desc
| limit 20

Real Example: API Gateway 4xx vs 5xx Analysis

fields @timestamp, path, status, errorMessage, requestId
| filter status >= 400
| stats 
    count() as error_count,
    count_distinct(requestId) as unique_requests
    by status, path
| sort error_count desc

ECS/Fargate Container Logs

ECS container logs capture application output from Docker containers.

ECS Log Stream Format

ecs/my-service/my-container/abc-123-456

Query ECS Container Logs

fields @timestamp, @message, @logStream
| parse @logStream /ecs\/(?<service>[^\/]+)\/(?<container>[^\/]+)\/(?<task_id>.*)/
| filter @message like /ERROR/
| stats count() by service, container
| sort count desc

ECS Task Failures

fields @timestamp, @message, @logStream
| filter @message like /error|exception|failed/
| parse @logStream /ecs\/(?<service>[^\/]+)/
| stats count() as error_count by service, bin(5m) as interval
| sort interval desc, error_count desc

ECS Container Resource Issues

fields @timestamp, @message
| filter @message like /memory|cpu|disk/
| parse @message /(?<metric_type>memory|cpu|disk)[^\d]*(?<value>[0-9.]+)/
| stats avg(value) by metric_type, bin(5m)

Real Example: ECS Service Health Check

fields @timestamp, @message, @logStream
| parse @logStream /ecs\/(?<service>[^\/]+)\/(?<container>[^\/]+)/
| filter @message like /health|ready|alive|status/
| parse @message /status[=:]?\s*(?<status>[a-zA-Z0-9]+)/
| stats 
    count() as checks,
    count(if(status like /healthy|ready|ok/, 1, 0)) as healthy_checks
    by service, bin(1m) as minute
| fields minute, service,
    healthy_checks * 100.0 / checks as health_pct
| filter health_pct < 100
| sort minute desc

RDS Database Logs

RDS provides multiple log types: error logs, slow query logs, and general logs.

RDS Error Log Queries

fields @timestamp, @message
| filter @message like /ERROR|WARNING|FATAL/
| parse @message /\[(?<severity>[A-Z]+)\]\s*(?<error_message>.*)/
| stats count() by severity
| sort count desc

RDS Slow Query Analysis

fields @timestamp, @message
| filter @message like /Query_time/
| parse @message /Query_time:\s*(?<query_time>[0-9.]+)\s*Lock_time:\s*(?<lock_time>[0-9.]+)/
| parse @message /Rows_sent:\s*(?<rows_sent>[0-9]+)\s*Rows_examined:\s*(?<rows_examined>[0-9]+)/
| filter query_time > 1.0
| sort query_time desc
| limit 50

Real Example: RDS Connection Issues

fields @timestamp, @message
| filter @message like /connection|client|disconnect/
| parse @message /(?<event_type>connection|disconnect|refused)/
| stats count() as occurrences by event_type, bin(5m) as interval
| sort interval desc, occurrences desc

VPC Flow Logs

VPC Flow Logs track network traffic for security and troubleshooting.

VPC Flow Log Format

2 123456789012 eni-abc123 192.168.1.1 10.0.0.1 443 52000 6 10 5000 1234567890 1234567891 ACCEPT OK

Format: version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status

Query VPC Flow Logs

fields @timestamp, @message
| parse @message /(?<version>\d+)\s+(?<account_id>\d+)\s+(?<interface_id>eni-[a-z0-9]+)\s+(?<src_ip>[\d.]+)\s+(?<dst_ip>[\d.]+)\s+(?<src_port>\d+)\s+(?<dst_port>\d+)\s+(?<protocol>\d+)\s+(?<packets>\d+)\s+(?<bytes>\d+)\s+(?<start>\d+)\s+(?<end>\d+)\s+(?<action>[A-Z]+)/
| filter action = "REJECT"
| stats count() as rejected_connections by src_ip, dst_ip, dst_port
| sort rejected_connections desc
| limit 50

VPC Traffic Volume by IP

fields @message
| parse @message /(?<src_ip>[\d.]+)\s+(?<dst_ip>[\d.]+)\s+\d+\s+\d+\s+\d+\s+\d+\s+(?<bytes>\d+)/
| stats sum(bytes) as total_bytes by src_ip, dst_ip
| fields src_ip, dst_ip, total_bytes / 1024 / 1024 as total_mb
| sort total_mb desc
| limit 20

Real Example: Security Analysis - Rejected Connections

fields @timestamp, @message
| parse @message /(?<interface_id>eni-[a-z0-9]+)\s+(?<src_ip>[\d.]+)\s+(?<dst_ip>[\d.]+)\s+(?<src_port>\d+)\s+(?<dst_port>\d+).*(?<action>REJECT)/
| filter action = "REJECT"
| stats 
    count() as attempts,
    count_distinct(src_port) as unique_src_ports
    by src_ip, dst_ip, dst_port, bin(5m) as interval
| filter attempts > 10
| sort interval desc, attempts desc

CloudTrail Logs

CloudTrail tracks AWS API calls for security, compliance, and troubleshooting.

CloudTrail Event Structure

{
  "eventTime": "2024-01-15T10:30:45Z",
  "eventName": "PutObject",
  "userIdentity": {
    "type": "IAMUser",
    "userName": "admin"
  },
  "sourceIPAddress": "192.168.1.1",
  "errorCode": "AccessDenied"
}

Query CloudTrail for Errors

fields @timestamp, userIdentity.userName, eventName, errorCode, sourceIPAddress
| filter ispresent(errorCode)
| stats count() as error_count by errorCode, eventName
| sort error_count desc

CloudTrail Unauthorized Attempts

fields @timestamp, userIdentity.userName, eventName, sourceIPAddress, errorMessage
| filter errorCode = "UnauthorizedOperation" or errorCode = "AccessDenied"
| stats count() as attempts by userIdentity.userName, eventName, sourceIPAddress
| sort attempts desc

Real Example: CloudTrail Security Monitoring

fields @timestamp, eventName, userIdentity.userName, sourceIPAddress, errorCode
| filter eventName in [
    "DeleteBucket", 
    "DeleteDBInstance", 
    "TerminateInstances",
    "DeleteTable",
    "DeleteFunction"
  ]
| stats 
    count() as deletion_attempts,
    count(if(ispresent(errorCode), 1, 0)) as failed_attempts,
    count(if(not ispresent(errorCode), 1, 0)) as successful_deletions
    by eventName, userIdentity.userName
| sort deletion_attempts desc

Application Load Balancer Logs

ALB logs provide detailed HTTP request/response data.

ALB Log Format

http 2024-01-15T10:30:45.123456Z app/my-alb/abc123 192.168.1.1:52000 10.0.0.1:8000 0.001 0.002 0.000 200 200 1234 567 "GET https://example.com:443/ HTTP/1.1" "Mozilla/5.0..." ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2

Query ALB Access Logs

fields @timestamp, @message
| parse @message /(?<type>[^ ]+) (?<time>[^ ]+) (?<elb>[^ ]+) (?<client_ip>[^ ]+):(?<client_port>[0-9]+) (?<target_ip>[^ ]+)[:-](?<target_port>[0-9]+|-) (?<request_processing_time>[-.0-9]+) (?<target_processing_time>[-.0-9]+) (?<response_processing_time>[-.0-9]+) (?<elb_status_code>|-?[0-9]+) (?<target_status_code>-|[0-9]+) (?<received_bytes>[0-9]+) (?<sent_bytes>[0-9]+) "(?<request_method>[A-Z]+) (?<request_url>[^ ]+) (?<request_protocol>[^"]+)"/
| filter elb_status_code >= 400
| stats count() by elb_status_code, request_method, request_url
| sort count desc

ALB Performance Analysis

fields @message
| parse @message /(?<client_ip>[^ ]+):(?<client_port>[0-9]+) (?<target_ip>[^ ]+)[:-](?<target_port>[0-9]+|-) (?<request_processing_time>[-.0-9]+) (?<target_processing_time>[-.0-9]+) (?<response_processing_time>[-.0-9]+) (?<elb_status_code>|-?[0-9]+)/
| fields 
    (request_processing_time + target_processing_time + response_processing_time) as total_time,
    target_processing_time,
    elb_status_code
| filter total_time > 0
| stats 
    count() as requests,
    avg(total_time) as avg_total_ms,
    avg(target_processing_time) as avg_target_ms,
    pct(total_time, 95) as p95_total_ms
    by elb_status_code
| sort avg_total_ms desc

Real Example: ALB Error Analysis

fields @timestamp, @message
| parse @message /"(?<method>[A-Z]+) (?<url>[^ ]+)[^"]*" "(?<user_agent>[^"]+)".*(?<elb_status>[0-9]{3}) (?<target_status>[0-9]{3}|-)/
| filter elb_status >= 400 or target_status >= 400
| stats 
    count() as error_count,
    count_distinct(url) as unique_urls
    by elb_status, target_status
| sort error_count desc

ElastiCache Logs

ElastiCache (Redis/Memcached) logs engine events and errors.

Query ElastiCache Slow Logs

fields @timestamp, @message
| filter @message like /slow/
| parse @message /slow.*took (?<duration>[0-9]+) microseconds/
| stats 
    count() as slow_commands,
    avg(duration) / 1000 as avg_ms,
    max(duration) / 1000 as max_ms

ElastiCache Connection Issues

fields @timestamp, @message
| filter @message like /connection|timeout|error/
| parse @message /(?<event_type>connection|timeout|error|failed)/
| stats count() by event_type, bin(5m)

CodeBuild Logs

CodeBuild logs contain build output and errors.

Query CodeBuild Failures

fields @timestamp, @message
| filter @message like /ERROR|FAILED|failed/
| parse @message /ERROR:\s*(?<error_message>.*)/
| stats count() by error_message
| sort count desc

CodeBuild Phase Duration

fields @timestamp, @message
| parse @message /Phase: (?<phase>[A-Z_]+) Status: (?<status>[A-Z]+) Duration: (?<duration>[0-9]+) seconds/
| stats avg(duration) as avg_seconds by phase, status
| sort avg_seconds desc

Step Functions Logs

Step Functions logs state machine executions.

Query Step Functions Failures

fields @timestamp, @message
| filter @message like /TaskFailed|ExecutionFailed|TaskTimedOut/
| parse @message /"error":"(?<error>[^"]+)"/
| parse @message /"cause":"(?<cause>[^"]+)"/
| stats count() by error
| sort count desc

Cross-Service Correlation Patterns

Pattern: Trace Request Across Services

# Use X-Ray trace ID or custom request ID
fields @timestamp, @message, @log
| parse @message /(?:requestId|traceId|request_id)[=:]?\s*(?<request_id>[a-f0-9\-]{20,})/
| filter ispresent(request_id)
| sort @timestamp
| limit 100

Pattern: API Gateway → Lambda → RDS

Query multiple log groups:

1. API Gateway: /aws/apigateway/my-api

2. Lambda: /aws/lambda/my-function

3. RDS: /aws/rds/instance/my-db/slowquery

fields @timestamp, @log, @message
| parse @message /requestId[=:]?\s*(?<req_id>[a-f0-9\-]+)/
| filter ispresent(req_id)
| stats count() by req_id, @log
| sort @timestamp

Service-Specific Best Practices

Lambda Best Practices

1. Always include request ID in logs

2. Use structured logging (JSON)

3. Monitor cold starts separately

4. Track memory usage trends

API Gateway Best Practices

1. Enable execution logging (not just access logging)

2. Include correlation IDs

3. Monitor 4xx vs 5xx separately

4. Track latency by endpoint

RDS Best Practices

1. Enable slow query logs (set threshold appropriately)

2. Monitor connection pool exhaustion

3. Track long-running queries

4. Alert on replication lag

ECS Best Practices

1. Use structured logging in containers

2. Include container ID in logs

3. Monitor task recycling frequency

4. Track resource utilization

Key Takeaways

• Each AWS service has unique log formats and structures

• Lambda logs include automatic START/END/REPORT entries

• API Gateway requires explicit log enablement

• VPC Flow Logs use space-delimited format

• CloudTrail logs all API calls for security auditing

• ALB logs provide detailed HTTP metrics

• Use parsing to extract service-specific fields

• Correlate across services using request/trace IDs

• Enable appropriate log levels for each service

In Part 5, we'll build comprehensive observability dashboards using CloudWatch, combining these queries into actionable visualizations for real-time monitoring.