Intelligent Data Analysis for Observability: Using AI to Surface Root Causes from Logs, Metrics, and Traces
A practical guide to going beyond dashboards, using Python and Amazon Bedrock to turn raw telemetry volume into plain language root cause summaries during an incident
Why This Topic
AWS Builder Center content that performs well shares a consistent shape: a real implementation experience, a step by step build, an explained architecture decision, and lessons pulled from an actual pain point, not abstract theory. Observability content on the platform is often about collection, getting metrics, logs, and traces into a backend. Far less content addresses what happens after collection: an engineer staring at millions of log lines and thousands of spans during an active incident, trying to manually find the needle.
Intelligent data analysis for observability is the natural next step once telemetry collection is solved. It applies AI, specifically log clustering, embedding based similarity search, and LLM summarization, directly to the volume problem that dashboards alone don't solve. This sits precisely at your niche, Observability Engineering and DevOps Automation with Python, extended into the analysis layer that turns collected telemetry into an actual answer during an incident, not just a place to look for one.
The Problem: Collection Solved, Analysis Still Manual
Most mature engineering orgs have solved telemetry collection. Logs flow into a centralized store, metrics into Prometheus or CloudWatch, traces into Tempo or X-Ray. What remains stubbornly manual is the analysis step during an actual incident:
- An engineer greps or filters through tens of thousands of log lines by eye, looking for the one pattern that matters
- Nobody remembers that a nearly identical incident happened four months ago, so the same root cause gets rediscovered from scratch
- Correlating a spike in one metric with a specific error pattern in logs and a specific slow span in traces still happens manually, one tab at a time
Dashboards are excellent at showing that something is wrong. They are far weaker at explaining why, especially at 2am, under time pressure, with a service nobody on call fully remembers the internals of.
The Architecture: From Raw Telemetry to a Root Cause Summary
[ Logs ] [ Metrics ] [ Traces ] | | | v v v[ Log embedding [ Anomaly window [ Slow/error span and clustering ] detection ] extraction ] | | | +--------+--------+--------+---------+ | v [ Correlation window: same service, same time range ] | v [ Retrieval: similar past incidents via vector search ] | v [ Bedrock: synthesize into plain-language root cause summary ] | v [ Incident channel: summary + supporting evidence links ]The core idea: use lightweight statistical and embedding based techniques to narrow petabytes of telemetry down to the handful of log lines, spans, and metric windows that actually matter, then use an LLM only at the final synthesis step, where its language ability adds real value rather than trying to make it reason over raw volume directly.
Step 1: Clustering Logs to Reduce Volume Before Analysis
Feeding raw logs into an LLM doesn't scale, and it's the wrong tool for the job anyway. The first step is reducing volume through clustering, so only distinct, representative patterns move forward:
python
import boto3import jsonimport numpy as npfrom sklearn.cluster import DBSCAN
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
def get_embedding(text): response = bedrock.invoke_model( modelId="amazon.titan-embed-text-v2:0", body=json.dumps({"inputText": text[:2000]}), ) result = json.loads(response["body"].read()) return result["embedding"]
def cluster_log_lines(log_lines, eps=0.3, min_samples=3): """Groups similar log lines together and returns one representative example per cluster, drastically reducing volume for downstream analysis.""" embeddings = np.array([get_embedding(line) for line in log_lines])
clustering = DBSCAN(eps=eps, min_samples=min_samples, metric="cosine") labels = clustering.fit_predict(embeddings)
clusters = {} for label, line in zip(labels, log_lines): clusters.setdefault(label, []).append(line)
representatives = [] for label, lines in clusters.items(): representatives.append({ "cluster_id": int(label), "count": len(lines), "example": lines[0], "is_noise": label == -1, })
return sorted(representatives, key=lambda c: c["count"], reverse=True)A 50,000 line log dump for a service during an incident window typically reduces to 15 to 40 distinct clusters. That's the actual volume problem being solved here, not summarizing text, but reducing millions of near duplicate lines down to a small number of genuinely distinct patterns worth a human or an LLM looking at.
Step 2: Retrieving Similar Past Incidents
Once distinct patterns are identified, checking whether something similar has happened before turns tribal knowledge into a queryable resource:
python
import boto3import json
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")opensearch = boto3.client("opensearchserverless") # or your vector store of choice
def find_similar_past_incidents(cluster_example, top_k=3, vector_index_client=None): """Embeds the current log pattern and searches a vector store of past incident postmortems for similar prior occurrences.""" query_embedding = get_embedding(cluster_example)
# vector_index_client wraps your chosen vector store (OpenSearch, # Amazon Kendra, or a simple in-memory index for smaller teams) results = vector_index_client.search( vector=query_embedding, top_k=top_k, index_name="incident-postmortems", )
return [ { "incident_id": r["metadata"]["incident_id"], "summary": r["metadata"]["summary"], "resolution": r["metadata"]["resolution"], "similarity_score": r["score"], } for r in results ]Populating this index is a one time bootstrap: embed the summary section of each past postmortem document as it's written, and the retrieval step above starts paying off from the very next incident onward.
Step 3: Synthesizing a Root Cause Summary
This is the step where an LLM genuinely earns its place: taking the narrowed down evidence, correlated metric window, error span details, and similar past incidents, and turning it into a coherent, readable summary an on-call engineer can act on immediately.
python
def generate_root_cause_summary(service_name, log_clusters, metric_anomaly, trace_evidence, similar_incidents): evidence_block = "\n".join( f"- {c['count']}x occurrences: {c['example'][:200]}" for c in log_clusters[:5] ) similar_block = "\n".join( f"- Incident {i['incident_id']} (similarity {i['similarity_score']:.2f}): {i['summary']} | Resolved by: {i['resolution']}" for i in similar_incidents )
prompt = f"""You are assisting an on-call engineer during an active incident for the service "{service_name}".
Metric anomaly detected: {metric_anomaly}
Top log patterns observed during the incident window:{evidence_block}
Slowest/error trace evidence: {trace_evidence}
Similar past incidents found:{similar_block}
Write a concise incident summary in 4 to 6 sentences: what is likely happening, what evidence supports that, and one concrete first troubleshooting step, referencing the similar past incident's resolution if it appears relevant."""
response = bedrock.invoke_model( modelId="anthropic.claude-3-5-sonnet-20241022-v2:0", body=json.dumps({ "anthropic_version": "bedrock-2023-05-31", "max_tokens": 400, "messages": [{"role": "user", "content": prompt}], }), ) result = json.loads(response["body"].read()) return result["content"][0]["text"]Notice the design boundary here: the LLM is given already narrowed, already correlated evidence, it is not asked to search raw telemetry itself. This keeps the system fast, cost effective, and auditable, an engineer can always trace the summary back to the exact log clusters and metric window that produced it.
Real World Scenario: What This Surfaces
During a checkout service incident, 40,000 log lines from the affected window cluster down to 22 distinct patterns in under a minute. One cluster, occurring 340 times, matches an error pattern with 0.89 similarity to a postmortem from five months earlier involving a downstream payment provider timeout. The generated summary tells the on-call engineer, within the first two minutes of the page, that the current error pattern closely resembles a prior payment provider timeout incident, and that the previous resolution involved a circuit breaker configuration change, giving them a concrete first thing to check instead of starting the investigation from zero.
Common Pitfalls
- Feeding raw, unclustered logs directly to an LLM. This is slow, expensive, and unreliable at volume. Cluster and reduce first, synthesize last.
- Skipping the postmortem embedding index. The retrieval step is what turns this from "AI summarizes today's logs" into "AI remembers what the team learned five months ago." Without it, you lose the most valuable part of the pattern.
- Treating the LLM summary as ground truth. It is a starting hypothesis built from real evidence, not a verified diagnosis. Always link back to the underlying clusters and traces so engineers can verify, not just trust.
- Ignoring embedding model consistency. Use the same embedding model for both log clustering and incident retrieval, mixing embedding models breaks the similarity comparisons silently.
- Running this only during incidents. Clustering and retrieval work just as well for post-incident review and for spotting recurring low-grade issues before they escalate into a full incident.
Closing Thought
Intelligent data analysis for observability isn't about replacing dashboards, it's about acknowledging that dashboards answer "what" while engineers still need help answering "why," especially at telemetry volumes no human can manually scan during a live incident. The pattern here, cluster to reduce volume, retrieve to recall institutional knowledge, synthesize to communicate clearly, keeps AI in the role it's actually good at: turning already-narrowed evidence into a fast, readable starting point, while keeping the underlying detection and correlation logic deterministic and auditable. For teams that have already invested in telemetry collection, this analysis layer is often the highest leverage next step, the difference between having the data and actually being able to use it under pressure.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article