Part 1: Introduction to CloudWatch Logs Insights

My CloudWatch Logs Journey

I remember my first production incident in AWS - errors were happening, customers were complaining, and I was frantically scrolling through raw CloudWatch logs trying to find the needle in the haystack. That's when I discovered CloudWatch Logs Insights, and it completely changed how I approach log analysis in AWS.

In this series, I'll share everything I've learned about CloudWatch Logs Insights from building and maintaining production AWS infrastructure.

What is CloudWatch Logs Insights?

CloudWatch Logs Insights is AWS's purpose-built query language for searching and analyzing log data. It's not just a log viewer - it's a powerful analytics engine that helps you extract meaningful insights from millions of log events in seconds.

Why CloudWatch Logs Insights Matters

From my experience, here's why it's essential:

1. Speed: Query millions of log events in seconds

2. Cost-effective: Pay only for queries you run (no index fees)

3. Integrated: Native integration with all AWS services

4. Powerful: Rich query language with aggregations and visualizations

5. Real-time: Query logs as they arrive

Understanding CloudWatch Logs Structure

Before writing queries, let me explain how CloudWatch Logs is organized.

Log Hierarchy

AWS Account
  └── Log Groups (e.g., /aws/lambda/my-function)
      └── Log Streams (e.g., 2024/01/15/[$LATEST]abc123)
          └── Log Events (individual log entries)

Log Groups: Logical containers for logs from the same source Log Streams: Sequences of log events from a single source Log Events: Individual log entries with timestamp and message

Common Log Groups I Work With

/aws/lambda/my-function           # Lambda function logs
/aws/apigateway/my-api            # API Gateway logs
/ecs/my-cluster/my-service        # ECS container logs
/aws/rds/instance/my-db/error     # RDS error logs
/aws/codebuild/my-project         # CodeBuild logs
my-application/production         # Custom application logs

Setting Up CloudWatch Logs

Let me walk you through enabling logs for AWS services.

Lambda Function Logs

Lambda automatically creates log groups, but you need proper IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

API Gateway Access Logs

Enable API Gateway logs (this took me a while to figure out):

# Using AWS CLI
aws apigateway update-stage \
  --rest-api-id abc123 \
  --stage-name prod \
  --patch-operations \
    op=replace,path=/accessLogSettings/destinationArn,value=arn:aws:logs:us-east-1:123456789012:log-group:/aws/apigateway/my-api \
    op=replace,path=/accessLogSettings/format,value='$context.requestId'

ECS Container Logs

Configure ECS task definition:

{
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/ecs/my-service",
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "ecs"
    }
  }
}

Accessing CloudWatch Logs Insights

Via AWS Console

1. Navigate to CloudWatch → Logs → Insights

2. Select log group(s) to query

3. Write your query

4. Set time range

5. Run query

Via AWS CLI

# Run a query
aws logs start-query \
  --log-group-name /aws/lambda/my-function \
  --start-time $(date -u -d '1 hour ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20'

# Get query results
aws logs get-query-results --query-id <query-id>

Via AWS SDK (TypeScript)

import { CloudWatchLogsClient, StartQueryCommand, GetQueryResultsCommand } from "@aws-sdk/client-cloudwatch-logs";

const client = new CloudWatchLogsClient({ region: "us-east-1" });

// Start query
const startQuery = await client.send(new StartQueryCommand({
  logGroupName: "/aws/lambda/my-function",
  startTime: Math.floor((Date.now() - 3600000) / 1000),
  endTime: Math.floor(Date.now() / 1000),
  queryString: `
    fields @timestamp, @message
    | filter @message like /ERROR/
    | sort @timestamp desc
    | limit 20
  `
}));

// Poll for results
const getResults = await client.send(new GetQueryResultsCommand({
  queryId: startQuery.queryId
}));

Your First CloudWatch Logs Insights Query

Let me walk you through your first query. This is the first one I ran after discovering Insights.

Basic Query Structure

CloudWatch Logs Insights queries follow this pattern:

fields <field1>, <field2>
| filter <condition>
| stats <aggregation>
| sort <field> desc/asc
| limit <number>

Example: Finding Errors

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 20

Breaking it down:

1. fields @timestamp, @message - Select which fields to display

2. filter @message like /ERROR/ - Find log events containing "ERROR"

3. sort @timestamp desc - Sort by timestamp, newest first

4. limit 20 - Return only 20 results

Automatic Fields

CloudWatch provides these automatic fields for all logs:

• @timestamp - Event timestamp

• @message - Log message content

• @logStream - Log stream name

• @log - Log group identifier

• @ptr - Internal pointer (rarely used)

Running Your First Query

1. Go to CloudWatch Logs Insights

2. Select /aws/lambda/your-function log group

3. Paste this query:

fields @timestamp, @message
| sort @timestamp desc
| limit 10

1. Set time range to "Last 1 hour"

2. Click "Run query"

You should see the 10 most recent log events!

Understanding Query Results

Results are displayed as:

• Table view: Columnar format (default)

• Log view: Raw log format

• Visualization: For aggregated queries

Result Fields

Each result row contains:

• Selected fields from your query

• Automatic fields (@timestamp, @message, etc.)

• Parsed fields (if using parse command)

Common Query Patterns I Use Daily

Pattern 1: Find Errors in Last Hour

fields @timestamp, @message
| filter @message like /ERROR|WARN/
| sort @timestamp desc

Pattern 2: Count Events by Type

fields @message
| stats count() by @logStream
| sort count desc

Pattern 3: Find Slow Requests

fields @timestamp, @message
| filter @message like /duration/
| parse @message /Duration: (?<duration>[0-9.]+) ms/
| filter duration > 1000
| sort duration desc

Query Cost and Limits

Understanding costs helped me optimize my queries:

Pricing (as of my experience)

• Query pricing: $0.005 per GB of data scanned

• Log storage: $0.50 per GB per month

• Data ingestion: $0.50 per GB

Query Limits

• Time range: Up to 366 days

• Query timeout: 15 minutes

• Results: Up to 10,000 rows displayed (though millions can be scanned)

• Concurrent queries: 30 per account per region

Cost Optimization Tips

From my experience:

1. Use specific time ranges - Don't query more data than needed

2. Filter early - Apply filters before aggregations

3. Select specific fields - Don't use fields @* unnecessarily

4. Use log retention policies - Delete old logs

5. Sample data during development - Use limit while testing

CloudWatch Logs Insights vs Other Options

Let me share when I use Logs Insights vs alternatives:

CloudWatch Logs Insights

• Best for: Real-time troubleshooting, adhoc queries

• Cost: Pay per query

• Speed: Very fast for recent data

Athena + S3

• Best for: Long-term analysis, complex joins

• Cost: Pay per query + S3 storage

• Speed: Slower, but handles larger datasets

OpenSearch/Elasticsearch

• Best for: Full-text search, complex queries

• Cost: Instance costs + storage

• Speed: Fast with proper indexing

When I Use Each

• Troubleshooting production issue: Logs Insights

• Analyzing 6 months of data: Athena

• Security log analysis: OpenSearch

• Real-time monitoring: Logs Insights + CloudWatch Alarms

Time Ranges and Filters

Time is critical in log analysis. Here's how I work with time:

Relative Time Ranges

Last 5 minutes
Last 30 minutes
Last 1 hour
Last 3 hours
Last 12 hours
Last 1 day
Last 3 days
Last 1 week

Absolute Time Ranges

Select custom start and end times for specific incident analysis.

Time-Based Filtering

# Events in specific time range
fields @timestamp, @message
| filter @timestamp >= "2024-01-15T10:00:00" and @timestamp < "2024-01-15T11:00:00"

Practical Tips from My Experience

Tip 1: Start Simple

Begin with basic queries and add complexity:

# Step 1: See recent logs
fields @timestamp, @message
| sort @timestamp desc
| limit 10

# Step 2: Add filtering
fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc

# Step 3: Add parsing and aggregation
fields @timestamp, @message
| filter @message like /ERROR/
| parse @message /Error: (?<errorType>[A-Za-z]+)/
| stats count() by errorType

Tip 2: Save Queries

Save frequently-used queries in CloudWatch:

• Click "Save" after writing a query

• Give it a descriptive name

• Access from "Saved queries" tab

Tip 3: Use Query History

CloudWatch remembers your recent queries:

• Access via "History" tab

• Quickly re-run previous queries

• Learn from past successes

Tip 4: Test with Smaller Time Ranges

When developing queries:

1. Use last 15 minutes of data

2. Verify query works

3. Expand time range as needed

Tip 5: Understand Your Log Format

Before querying, look at raw logs to understand:

• Field names and structure

• Log format (JSON, plaintext, etc.)

• Common patterns to search for

Sample Queries for Learning

Try these queries on your AWS logs:

Query 1: Latest Log Events

fields @timestamp, @message, @logStream
| sort @timestamp desc
| limit 20

Query 2: Count by Log Stream

stats count() by @logStream
| sort count desc

Query 3: Find Specific Text

fields @timestamp, @message
| filter @message like /timeout/
| sort @timestamp desc

Query 4: Time Bucketing

stats count() by bin(5m)

Common Mistakes I Made

Let me save you time by sharing my mistakes:

Mistake 1: Querying Too Much Data

# Bad - Queries entire log group
fields @timestamp, @message

# Good - Specific time range
fields @timestamp, @message
| filter @timestamp > ago(1h)

Mistake 2: Not Using Filters Early

# Less efficient
fields @timestamp, @message
| parse @message /duration=(?<dur>[0-9]+)/
| filter dur > 1000

# More efficient
fields @timestamp, @message
| filter @message like /duration/
| parse @message /duration=(?<dur>[0-9]+)/
| filter dur > 1000

Mistake 3: Forgetting to Sort

# Results in random order
fields @timestamp, @message
| limit 20

# Properly sorted
fields @timestamp, @message
| sort @timestamp desc
| limit 20

Key Takeaways

• CloudWatch Logs Insights is AWS's native log analysis engine

• Queries are fast and cost-effective for recent data

• Log groups organize logs from similar sources

• Basic query structure: fields → filter → stats → sort → limit

• Always use specific time ranges to control costs

• Start simple and add complexity incrementally

• Save frequently-used queries for reuse

In Part 2, we'll dive deep into CloudWatch query syntax, commands, and operators that I use every day for log analysis.