AWS Builder Center

AI-Powered Security Observability: Detecting Anomalous Behavior in DevOps Pipelines with Python and AWS

A practical guide to combining observability telemetry with machine learning to catch suspicious pipeline and runtime behavior before it becomes a breach.

Google cloud certified professional machine learning engineer

Why This Topic

AWS Builder Center content that performs well shares a consistent shape: a real implementation experience, a step by step build, an explained architecture decision, and lessons pulled from an actual pain point, not abstract theory. Security content on the platform tends to fall into two separate buckets: traditional security tooling (GuardDuty, IAM policy review, WAF rules) on one side, and observability tooling (metrics, logs, traces) on the other. Very little content connects the two, even though the connection is exactly where modern threat detection is heading.
AI-Powered Security Observability closes that gap. It applies the same telemetry pipelines DevOps teams already build for reliability (logs, metrics, traces) to a security use case: using statistical and ML-based anomaly detection to flag suspicious behavior in CI/CD pipelines and running workloads, long before a rule-based alert would trigger.
This sits precisely at your niche, Observability Engineering and DevOps Automation with Python, extended into the security domain that's becoming a natural next step for teams already investing in telemetry infrastructure.



The Problem: Rule-Based Security Alerts Miss Novel Behavior

Most security monitoring today is rule based: alert if an IAM role is used outside business hours, alert if a deployment happens from an unrecognized IP, alert if a container makes an outbound connection to a new domain. Rules catch known patterns well, but they share the same blind spot: they only catch what someone already thought to write a rule for.
A compromised CI/CD credential that deploys during normal business hours, from a known IP, using a legitimate pipeline, will sail past every rule based check while still being a genuine compromise if the behavior is subtly different from that pipeline's normal pattern (unusual number of resources touched, an unfamiliar deployment target, a spike in API calls per run).
This is precisely the class of problem anomaly detection is built for: not "did this violate a known rule" but "does this look different from how this system normally behaves."



The Architecture: Telemetry In, Anomaly Scoring Out

[ CI/CD Pipeline Runs ] [ Runtime Workloads ]
| |
(OTel instrumentation) (CloudTrail + VPC Flow Logs)
| |
v v
[ OTel Collector ] <---------- [ Log/Event Aggregation ]
|
v
[ Feature Extraction (Python): calls per run, resource count,
deploy time-of-day, source IP entropy, API call sequence ]
|
v
[ Anomaly Detection Model: Isolation Forest / rolling z-score ]
|
score > threshold?
|
+----+----+
| |
no yes
| |
[ log ] [ Bedrock-assisted triage summary -> SNS/Slack alert ]
The key architectural idea: observability data is already the raw material for security detection. You don't need a separate telemetry pipeline for security, you need a feature extraction and scoring layer on top of what you're likely already collecting for reliability purposes.



Step 1: Extracting Behavioral Features from Pipeline Telemetry

Building on OTel-instrumented pipeline traces (the same kind covered in pipeline observability work), each pipeline run can be converted into a feature vector describing its behavior:
python
import boto3
from datetime import datetime
import numpy as np

cloudtrail = boto3.client("cloudtrail")

def extract_run_features(pipeline_run_id, start_time, end_time):
"""Pulls CloudTrail events tied to a pipeline execution and
converts them into a numeric feature vector."""
events = []
paginator = cloudtrail.get_paginator("lookup_events")
for page in paginator.paginate(
LookupAttributes=[
{"AttributeKey": "ReadOnly", "AttributeValue": "false"}
],
StartTime=start_time,
EndTime=end_time,
):
events.extend(page["Events"])

api_call_count = len(events)
unique_resources = len({e.get("Resources", [{}])[0].get("ResourceName", "")
for e in events if e.get("Resources")})
unique_source_ips = len({e.get("SourceIPAddress", "") for e in events})
hour_of_day = start_time.hour

return np.array([
api_call_count,
unique_resources,
unique_source_ips,
hour_of_day,
])
This turns a pipeline run from an opaque pass or fail event into a structured behavioral fingerprint: how many API calls it made, how many distinct resources it touched, how many source IPs were involved, and what time of day it ran.



Step 2: Scoring Behavior with Isolation Forest

Isolation Forest is a strong fit here because it doesn't require labeled attack data, which almost no team has. It learns what "normal" looks like from historical runs and scores new runs by how easily they can be isolated from that baseline:
python
from sklearn.ensemble import IsolationForest
import numpy as np
import joblib

def train_baseline_model(historical_feature_vectors, model_path="pipeline_baseline.joblib"):
"""Trains on at least a few weeks of normal pipeline runs."""
X = np.array(historical_feature_vectors)
model = IsolationForest(
n_estimators=200,
contamination=0.02, # assume roughly 2% of runs may be anomalous
random_state=42,
)
model.fit(X)
joblib.dump(model, model_path)
return model

def score_run(model, feature_vector):
"""Returns True if anomalous, plus a normalized anomaly score."""
prediction = model.predict([feature_vector])[0] # -1 = anomaly, 1 = normal
raw_score = model.decision_function([feature_vector])[0] # lower = more anomalous
is_anomalous = prediction == -1
return is_anomalous, raw_score
The  contamination  parameter matters more than it looks. Setting it too high (say 10%) trains the model to treat unusual behavior as routine, quietly raising your false negative rate. Setting it too low chokes the pipeline with false positives. Two percent is a reasonable starting point for CI/CD pipeline runs, but it should be tuned against your own historical run data, not copied blindly.



Step 3: AI-Assisted Triage for Flagged Runs

A raw anomaly score isn't actionable on its own, someone still has to figure out why a run got flagged. This is where a lightweight call to a foundation model (via Amazon Bedrock) adds real value: turning a numeric anomaly into a plain-language triage summary for the on-call engineer.
python
import boto3
import json

bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")

def generate_triage_summary(pipeline_run_id, feature_vector, baseline_stats, anomaly_score):
prompt = f"""A CI/CD pipeline run was flagged as anomalous by a statistical model.

Run ID: {pipeline_run_id}
Anomaly score: {anomaly_score:.3f} (lower is more anomalous)
Observed features: API calls={feature_vector[0]}, unique resources={feature_vector[1]}, unique source IPs={feature_vector[2]}, hour of day={feature_vector[3]}
Typical baseline for this pipeline: API calls~{baseline_stats['api_calls_mean']:.0f}, unique resources~{baseline_stats['resources_mean']:.0f}

In 3 sentences, summarize what looks unusual about this run and suggest one concrete next step for the on-call engineer to check first."""

response = bedrock.invoke_model(
modelId="anthropic.claude-3-5-sonnet-20241022-v2:0",
body=json.dumps({
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 300,
"messages": [{"role": "user", "content": prompt}],
}),
)
result = json.loads(response["body"].read())
return result["content"][0]["text"]
This step is deliberately narrow in scope: the model summarizes and suggests, it doesn't decide or act. The anomaly detection and the decision to alert stay fully in the deterministic Python layer above. This keeps the system auditable, a security reviewer can always trace exactly which numeric threshold triggered an alert, with the AI layer only responsible for making that alert faster to triage.



Real World Scenario: What This Catches

A pipeline that normally makes around 40 API calls and touches 6 to 8 AWS resources per deployment suddenly runs with 340 API calls and touches 52 distinct resources, still using valid, non-expired credentials, still during business hours. A rule based system sees nothing wrong: valid credentials, normal hours, known IP. The Isolation Forest model, trained on that pipeline's actual historical behavior, flags it immediately because the feature vector is far outside the learned baseline. The Bedrock triage summary tells the on-call engineer, in plain language, that resource count and API call volume are roughly 8x the pipeline's typical pattern, and suggests checking recently modified deployment scripts first, turning a 20 minute manual investigation into a 2 minute confirmed lead.



Common Pitfalls

  • Training on too little history. Isolation Forest needs enough historical runs to learn what normal actually looks like, a few days of data will produce a noisy, unreliable baseline. Aim for several weeks minimum.
  • Mixing pipelines in one model. A deployment pipeline and a nightly data sync job have very different normal behavior. Train separate baseline models per pipeline type rather than one model for everything.
  • Treating anomaly detection as a replacement for rule based alerts. The two are complementary, rules catch known bad patterns fast and cheaply, anomaly detection catches the unknown ones. Keep both running.
  • Letting the LLM layer make decisions. Keep the alerting threshold and the anomaly score deterministic and auditable. Use the model for summarization and triage assistance only, not as the trigger itself.
  • Not retraining the baseline periodically. Pipelines evolve, dependency counts grow, deployment targets change. A baseline trained six months ago will start flagging normal growth as anomalous. Retrain on a regular cadence.



Closing Thought

AI-Powered Security Observability isn't about replacing existing security tooling, it's about extracting more value from telemetry teams are often already collecting for reliability purposes. The shift is conceptual as much as technical: instead of asking "did this violate a rule," ask "does this look like how this system normally behaves." For DevOps and platform teams that have already invested in observability pipelines, adding a behavioral anomaly layer is a comparatively small lift with a genuinely different category of detection to show for it, one that catches the threats nobody thought to write a rule for.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article