Beginner's Guide to AWS CloudWatch Logs & Insights
When your Lambda function fails with an unhandled exception or an API Gateway returns a 502 Bad Gateway, where do you look? Many junior engineers open the AWS Console, click into the CloudWatch log stream, and spend 15 minutes manually scrolling through hundreds of lines of raw text. There is a much faster, industry-standard way to find root causes: CloudWatch Logs Insights. Here is how to set up structured logging and query production errors in seconds.
Step 1: Output Structured JSON Logs
CloudWatch automatically indexes JSON fields. If you log plain strings, you are forced to rely on slow substring matches. If you log JSON dictionaries, CloudWatch allows you to filter and aggregate specific keys.
In your AWS Lambda handler (Python example):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import json
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
try:
user_id = event.get("queryStringParameters", {}).get("userId")
if not user_id:
logger.warning(json.dumps({
"level": "WARN",
"message": "Missing required userId parameter",
"requestId": context.aws_request_id
}))
return {"statusCode": 400, "body": "userId is required"}
# Simulate normal processing
logger.info(json.dumps({
"level": "INFO",
"message": "User lookup succeeded",
"userId": user_id,
"requestId": context.aws_request_id
}))
return {"statusCode": 200, "body": f"Found user {user_id}"}
except Exception as exc:
logger.error(json.dumps({
"level": "ERROR",
"message": str(exc),
"requestId": context.aws_request_id
}))
return {"statusCode": 500, "body": "Internal server error"}
Step 2: Query Your Logs with CloudWatch Insights
Instead of opening individual log streams:
- Open the CloudWatch Console and click Logs Insights in the navigation pane.
- In the log group selector dropdown, select your Lambda function's log group (e.g.,
/aws/lambda/my-service-dev). - Set your time range (e.g., Last 1 hour).
Step 3: Run Essential Troubleshooting Queries
Paste these ready-to-use queries into the query editor to isolate issues quickly:
1. Surface all error-level logs with timestamps
1
2
3
4
fields @timestamp, @message
| filter level = "ERROR"
| sort @timestamp desc
| limit 20
2. Find slow executions approaching Lambda timeouts
1
2
3
4
filter @type = "REPORT"
| fields @timestamp, @duration, @maxMemoryUsed / 1000000 as memoryUsedMB
| sort @duration desc
| limit 10
3. Count errors grouped by user or route
1
2
3
filter level = "ERROR" or level = "WARN"
| stats count(*) as issueCount by message
| sort issueCount desc
Pro-Tip: Set Log Retention Policies Immediately
By default, AWS CloudWatch log groups retain log events Never expire (indefinitely). Over time, gigabytes of test logs will slowly accumulate and show up on your monthly invoice.
Run this quick command via the AWS CLI to cap retention at 7 or 14 days for sandbox environments:
1
2
3
aws logs put-retention-policy \
--log-group-name /aws/lambda/my-service-dev \
--retention-in-days 7
Over to You
Learning log querying tools early bridges the gap between writing code locally and operating reliable cloud services.
Builders: What is your go-to technique for monitoring serverless applications? Do you stick to CloudWatch Logs Insights, or do you stream your logs to an external dashboard? Let's discuss below!
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article