Real-Time Anomaly Detection Pipeline with Kinesis Data Streams and Amazon Bedrock
Learn how to ingest high-volume logs, batch and enrich events, summarize activity, detect anomalies with an LLM, and keep ingestion asynchronous for low latency. You'll also explore shard scaling, Enhanced Fan-Out, Firehose archiving to S3, and practical cost, throughput, and performance tradeoffs for production workloads.
Modern applications produce a constant firehose of logs and events, things like application errors, security events, infrastructure metrics, and user activity. Sifting through that volume manually to catch the handful of log lines that actually matter is impossible at scale. In this tutorial, you'll build a pipeline that streams log and event data through Amazon Kinesis Data Streams, uses AWS Lambda to batch and enrich records, and calls Amazon Bedrock (Claude) to summarize and flag anomalies in near real time, with results persisted to DynamoDB for a dashboard or alerting layer to consume.
By the end, you'll understand
- How Kinesis Data Streams captures and shards high volume log data
- How to wire a Lambda consumer to batch records and call Bedrock for summarization and anomaly detection
- How to keep the pipeline async so Bedrock's variable latency never blocks ingestion
- Practical shard scaling and cost or latency tradeoffs
1. Why Kinesis and Bedrock for log anomaly detection
Traditional log anomaly detection relies on static rules or regex pattern matching, which is brittle and blind to novel failure modes. Feeding batches of raw log lines to an LLM lets you catch semantically unusual events, like a new error type, an unusual sequence of actions, or a subtle security signal, without hand writing a rule for every case.
Kinesis is a good fit for ingestion here because it is a fully managed service built for capturing and storing high volume data streams in real time. It is also durable and elastic, replicating data across three Availability Zones within a region and scaling up or down based on throughput requirements without data loss. Typical put to get delay is under one second, so records are available for processing almost immediately after they are added to the stream.
The critical architectural decision, borrowed from real production patterns, is to decouple the hot ingestion path from the LLM call. One production write up on a similar Bedrock powered pipeline describes keeping the hot path (Kinesis to Lambda to Bedrock to DynamoDB) fully async and the serving path completely decoupled, so the user never waits for Bedrock to respond. We apply the same principle here. Log producers never wait on Claude's response, and a separate read path always serves the latest available summary.
2. Architecture overview
Two consumers read from the same stream, the Lambda function that talks to Bedrock, and Firehose, which archives raw records to S3 for later reprocessing or model evaluation. This fan out pattern is a standard Kinesis capability. Kinesis Data Streams integrates with Kinesis Data Firehose for near real time transformation and delivery of streaming data into an AWS data lake like Amazon S3, alongside Lambda for record processing and CloudWatch for metrics.
Request lifecycle
The second diagram shows what happens to a single window of log records, end to end, including the async handoff to Bedrock.
3. Setting up the Kinesis stream
Create the stream with enough shards to handle your expected throughput. Each shard supports roughly 1 MB per second, or 1,000 records per second, of write capacity, so size accordingly. In a comparable production pipeline, one shard was said to handle 1 MB per second write or 1,000 records per second, and at higher user or service counts you need to plan shard count carefully.
1
2
3
4
5
6
7
8
# Create a Kinesis data stream with 4 shards
aws kinesis create-stream \
--stream-name log-events \
--shard-count 4 \
--region us-east-1
# Wait for the stream to become active
aws kinesis wait stream-exists --stream-name log-events --region us-east-1Send a test record using the CLI to confirm the stream is accepting writes.
1
2
3
4
aws kinesis put-record \
--stream-name log-events \
--data "service=checkout-api,level=ERROR,msg=Timeout connecting to payment gateway" \
--partition-key checkout-api4. The Lambda consumer, batching and calling Bedrock
The Lambda function is triggered by an event source mapping against the stream. It groups incoming records into a short window, builds a prompt, and calls Bedrock's InvokeModel (or Converse) API to get back a structured summary and anomaly flag.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
import json
import time
import boto3
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("AnomalyResults")
MODEL_ID = "anthropic.claude-sonnet-5-v1:0"
def build_prompt(log_lines: list[str]) -> str:
joined = "\n".join(log_lines)
return (
"You are a log anomaly detector. Given the following log lines, "
"identify anything unusual (novel error types, repeated failures, "
"suspicious sequences). Respond in JSON with keys "
"anomaly_detected (bool), severity (low, medium, or high), summary (string).\n\n"
f"Logs:\n{joined}"
)
def lambda_handler(event, context):
log_lines = [json.loads(r["kinesis"]["data"])["msg"] for r in event["Records"]]
prompt = build_prompt(log_lines)
response = bedrock.invoke_model(
modelId=MODEL_ID,
body=json.dumps({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 512,
"messages": [{"role": "user", "content": prompt}],
}),
)
result = json.loads(response["body"].read())
analysis = json.loads(result["content"][0]["text"])
table.put_item(Item={
"window_id": str(int(time.time())),
"summary": analysis["summary"],
"anomaly_detected": analysis["anomaly_detected"],
"severity": analysis["severity"],
"ttl": int(time.time()) + 3600, # 1 hour TTL
})
return {"statusCode": 200}A few implementation notes worth internalizing before you deploy this.
- Latency is variable, not fixed. In a comparable production pipeline, Claude Sonnet typically responded in 1 to 4 seconds but could spike, and monitoring the InvokeModel duration in CloudWatch is what tells you whether that is affecting freshness.
- Use a TTL based result store, not a blocking read. In that same pipeline, the serving Lambda set a 1 hour TTL on each entry, and if Bedrock had not updated the cache within that window, the system fell back to a default, for us that means treating it as no new anomalies to report rather than an empty dashboard.
- Use Enhanced Fan Out if you attach multiple independent Lambda consumers to the same stream, since it is recommended when several consumers read the same stream, avoiding read throughput contention between them.
5. Handling scale, shard splitting
As log volume grows, a shard can become a bottleneck. Kinesis supports resharding without downtime.
1
2
3
4
5
# Split a shard to increase parallel throughput
aws kinesis split-shard \
--stream-name log-events \
--shard-to-split shardId-000000000000 \
--new-starting-hash-key 170141183460469231731687303715884105728Pair this with a CloudWatch alarm on IncomingBytes and WriteProvisionedThroughputExceeded so you reshard proactively rather than after you have started dropping or throttling writes.
6. Component comparison
Not every part of this pipeline does the same job. Here is how the pieces map to responsibilities.
| Component | Role in the pipeline | Failure behavior if unavailable |
|---|---|---|
| Kinesis Data Streams | Durable, ordered ingestion buffer for log events | Producers retry with backoff, no data loss within retention window |
| Lambda Consumer | Batches records, builds prompts, calls Bedrock | Batch retried per event source mapping retry policy |
| Amazon Bedrock (Claude) | Semantic summarization and anomaly classification | Stale but valid result served from DynamoDB via TTL fallback |
| Kinesis Data Firehose | Archives raw records to S3 for replay or training | Buffers and retries, does not block the Lambda path |
| DynamoDB | Stores latest anomaly result per window | Dashboard shows a no data state, does not affect ingestion |
| CloudWatch | Observability for shard throughput and Lambda/Bedrock latency | Pipeline continues, only visibility is lost |
7. Extending the pipeline
A few natural next steps once the basic pipeline is running.
- Route Firehose archived data older than N days back through Amazon Bedrock Knowledge Bases to build a retrieval layer over historical incidents.
- Swap the raw invoke_model call for a Bedrock Agent if you want the model to call tools, such as looking up a runbook or querying a metrics API, before producing its summary.
- If you need bidirectional, low latency voice or interactive sessions on top of this pipeline, for example an on call assistant, Bedrock AgentCore Runtime now supports this natively. WebRTC joined WebSocket as a bidirectional streaming protocol supported by AgentCore Runtime, optimized for real time media delivery where low latency is critical, with Amazon Kinesis Video Streams available as a managed TURN option for the WebRTC relay.
References
- AWS, Getting Started With Amazon Kinesis Data Streams, https://aws.amazon.com/kinesis/data-streams/getting-started/
- DataEngineeringWithAWS, Streaming Data into Action, Building a Scalable Real-Time Analytics Platform on AWS, https://dataengineeringwithaws.com/posts/aws-kinesis-emr-realtime-analytics/
- DZone, Building a Real-Time AI Agent with AWS, https://dzone.com/articles/real-time-ai-agent-aws
- AWS, Amazon Bedrock AgentCore Runtime adds WebRTC support for real-time bidirectional streaming, https://aws.amazon.com/about-aws/whats-new/2026/03/amazon-bedrock-webrtc/
- Jeevi Academy, Building a Real-Time Data Processing Pipeline with AWS Kinesis, https://www.jeeviacademy.com/building-a-real-time-data-processing-pipeline-with-aws-kinesis/
- SquareOps, What is AWS Kinesis, A Complete Guide to Real-Time Data Streaming in 2026, https://squareops.com/knowledge/what-is-aws-kinesis-a-complete-guide-to-real-time-data-streaming-in-2026/
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article