
Beyond the DAG Failure: How We Built AirGuard to Investigate Workflow Incidents
Airflow excels at orchestrating workflows and surfacing operational failures. However, explaining why a workflow failed often requires manually correlating evidence across multiple systems. AirGuard automates that investigation. Built with Amazon Bedrock AgentCore, it collects evidence across Airflow and AWS Telemetry, reconstructs incident timelines, and delivers deterministic root cause analysis with actionable remediation.
Abstract
Apache Airflow coordinates everything from ETL and analytics pipelines to ML workflows and business critical batch processes, but investigating a failure still means jumping across Airflow logs, CloudWatch, CloudTrail, and Cost Explorer by hand. As workflows become more distributed, failures rarely originate in a single service. A scheduler misconfiguration, permission change, or downstream dependency failure can ripple across compute, data, and cost simultaneously, and no single dashboard reconstructs that chain of events.
AirGuard is an agentic investigation platform that complements existing Airflow deployments without touching workflow definitions or replacing existing observability tooling. It automatically gathers telemetry across Airflow and connected AWS services, reconstructs operational timelines, correlates cross service evidence, and produces explainable root cause analyses through deterministic evidence evaluation. It integrates through failure callbacks, scheduled investigations, or manual requests, and separates AI driven investigation orchestration from deterministic evidence evaluation, so every root cause it produces is reproducible and auditable, not just plausible sounding.
The Problem: Correlation Is a Human Bottleneck
A single failed DAG run rarely tells the whole story. A task retry might be masking a worker running out of memory. A worker OOM might be downstream of a scaling policy that changed an hour earlier. That scaling change might trace back to a cost control Lambda, an IAM permission update, or a noisy neighbor process on shared infrastructure.
None of this is visible from any one dashboard. Airflow's UI shows task state, not infrastructure state. CloudWatch shows metrics, not workflow semantics. CloudTrail shows who changed what, not why it mattered to a DAG run three services away. Engineers end up doing the correlation work by hand, pulling logs, cross referencing timestamps, and reconstructing a timeline from memory and tribal knowledge of what usually causes this.
AirGuard's core bet is that this correlation work is not a creative task. It's a retrieval and reasoning task over a well defined evidence space, which makes it a good fit for an agentic system paired with a deterministic verification layer, rather than a dashboard or a black box LLM guess.
The Core Philosophy: Determinism over Hallucination
When building AI tools for DevOps and SRE workflows, trust is everything. If an LLM hallucinates a root cause, the engineer wastes hours chasing a ghost. One of the earliest architectural decisions in AirGuard was preventing the language model from generating the final root cause at all.
Large language models are excellent investigators. They decide what evidence to collect next, recognize patterns across telemetry, and adapt their investigation as new information becomes available. But operational root cause analysis demands reproducibility: the same incident should always produce the same conclusion. So AirGuard splits the two responsibilities:
- The Investigator (AgentCore Harness): an LLM acts as a detective, iteratively collecting evidence through investigation capabilities such as
get_task_logsorget_lambda_errors.
- The Evaluator (Deterministic Engine): a strict, rule based correlation engine takes that evidence, evaluates it against known failure signatures, and deduces the root cause without relying on LLM inference.
Architect's Note: The Detective vs. Judge Pattern. The AgentCore Harness acts as the Detective, iteratively collecting evidence. The Deterministic Investigation Engine acts as the Judge, evaluating that evidence against explicit operational rules. This separation lets AI improve the investigation process without ever becoming the source of operational truth.
The LLM decides where to look. It never decides what the answer is.

pipeline = (
IncidentContext(event)
.collect_with_agentcore()
.normalize_evidence()
.correlate_failure_signatures()
.rank_findings()
.generate_rca()
.notify_slack()
)
CAPABILITIES = {
"airflow_logs": GetTaskLogs(),
"cloudwatch": InspectMetrics(),
"cloudtrail": IAMAudit(),
"cost": AnalyzeCostSpike(),
"lambda": InspectLambda(),
"redis": QueueHealth()
}
agent = AgentCoreHarness(
capabilities=CAPABILITIES
)
Step 1: The Event-Driven Trigger
AirGuard moves away from the "chatbot" model. It operates entirely on events, integrating through push based webhooks rather than polling for failures.
Apache Airflow trigger. A standard
on_failure_callback at the DAG or Task level fires an HTTP POST to AirGuard's FastAPI endpoint ( /api/v1/webhooks/airflow ) the moment a task fails. AirGuard maps the raw payload into a strongly typed context:Capturing the exact
workflow_id and failed_node_id up front means the agent knows precisely which Airflow logs to pull and which downstream AWS resources correlate with that specific DAG.AWS anomaly trigger. AirGuard isn't limited to Airflow originated incidents. If AWS Cost Explorer detects a billing spike, it fires an SNS notification that AirGuard catches, creating a context with
trigger_source: AWS_COST_ANOMALY and investigating which DAGs ran during the spike window, for example checking whether a backfill storm caused the anomaly.On the backend, the webhook handler is intentionally thin. It translates the payload and immediately hands off:
# backend/api/v1/airflow_webhook.py@router.post("/webhook")async def handle_airflow_webhook( payload: AirflowWebhookPayload, background_tasks: BackgroundTasks, service: InvestigationService = Depends(get_investigation_service)): # 1. Translate the raw payload into a structured IncidentContext context = IncidentContext( workflow_id=payload.dag_id, workflow_execution_id=payload.run_id, failed_node_id=payload.task_id, execution_timestamp=payload.execution_date, severity=Severity.HIGH, trigger_source=TriggerSource.AIRFLOW_WEBHOOK, investigation_goal=InvestigationGoal.ROOT_CAUSE_ANALYSIS ) # 2. Queue the investigation in the background investigation_id = service.create_investigation( user_query=f"Automated trigger for {payload.dag_id}", incident_context=context ) background_tasks.add_task(service.execute_pipeline_async, investigation_id)
return {"status": "investigation_queued", "investigation_id": investigation_id}Architect's Note: Event-Driven by Design. Investigations should never delay production workflows. When Airflow reports an incident, AirGuard immediately acknowledges the request and investigates asynchronously. This keeps orchestration responsive while the investigation continues independently in the background, decoupling workflow execution from investigation execution so the platform adds no latency to existing pipelines.
Step 2: Agentic Evidence Collection
Once an investigation is triggered, AirGuard hands the incident context to the Amazon Bedrock AgentCore Harness, which orchestrates an iterative evidence collection process. Instead of treating the investigation as a single prompt response interaction, the harness maintains state across multiple reasoning steps, letting newly discovered evidence guide subsequent investigation decisions.
The workflow runs in four stages:
- Plan. Construct an investigation prompt and expose only the relevant investigation capabilities.
- Investigate. The harness selects a capability (e.g. retrieve Airflow logs, inspect CloudWatch metrics), the backend executes it, and the returned evidence is added back to the investigation context.
- Iterate. Based on the new evidence, the harness decides whether additional capabilities should be invoked, for example spotting a "Lambda timeout" mention in the logs and deciding to check CloudWatch next.
- Terminate. The investigation concludes once sufficient evidence has been collected, or a configurable iteration limit (capped at 5) is reached.
python
# backend/agent/agentcore_adapter.pydef _run_inline_tool_loop(self, harness_id: str, tool_names: List[str], request: InvestigationRequest): messages = [{"role": "user", "content": [{"text": self._build_prompt(request)}]}] collected_evidence = [] # The Iterative Investigation Loop for iteration in range(5): response = client.invoke_harness( harnessArn=harness_id, messages=messages, tools=self._build_inline_tool_definitions(tool_names) ) _, tool_calls = self._parse_harness_stream(response["stream"]) if not tool_calls: break # LLM has concluded its investigation # Execute tools locally (e.g., fetch Airflow logs, query CloudWatch) tool_results = [] for tc in tool_calls: evidence = self.tool_executor.execute_single(tc["name"], request) collected_evidence.append(evidence) tool_results.append({ "toolResult": {"toolUseId": tc["id"], "content": [{"text": str(evidence)}]} }) # Feed the evidence back into the context window messages.append({"role": "user", "content": tool_results}) return EvidenceBundleResult(evidence=collected_evidence)Why AgentCore?
Amazon Bedrock AgentCore provides a managed execution environment for long running, tool driven AI agents. Its support for stateful execution, dynamic tool orchestration, and iterative reasoning maps naturally onto AirGuard's investigation workflow. The harness decides which capability to invoke next, evaluates the evidence that comes back, and decides whether more investigation is needed, much like an experienced platform engineer working an incident, without AirGuard needing to build custom agent orchestration infrastructure from scratch.
AgentCore Capability | AirGuard Usage |
|---|---|
Stateful execution | Maintains investigation context across multiple reasoning steps |
Tool orchestration | Dynamically invokes investigation capabilities instead of a static workflow |
Iterative reasoning | Determines whether additional evidence is required before concluding |
Managed execution | Handles agent lifecycle and orchestration |
Architect's Note: The Harness Is an Orchestrator, Not a Decision Maker. AgentCore orchestrates the investigation. It does not determine the root cause. It maintains investigation state, selects capabilities, and decides when enough evidence has been collected. Once evidence collection is complete, the deterministic investigation engine takes over for the actual operational reasoning.
Capability-driven, not raw API access
Rather than exposing raw AWS APIs to the language model, AirGuard exposes a registry of typed investigation capabilities, such as Retrieve DAG History, Inspect Lambda Errors, and Analyze SageMaker Training Jobs, implemented as pure Python functions in
backend/tools/registry.py . The agent reasons about what it needs to investigate; the Python capabilities handle how the investigation is actually performed.Domain | Example Capabilities |
|---|---|
Apache Airflow | DAG runs, task instances, scheduler state, task logs, dependencies |
AWS Services | CloudWatch, CloudTrail, Lambda, SageMaker, Amazon S3, Cost Explorer |
Platform Services | PostgreSQL metadata, Redis broker, worker metrics |
Architect's Note: Capability-Driven Investigation. This abstraction keeps credentials isolated, simplifies input validation, and lets new services be integrated without touching investigation logic. The agent reasons about objectives, not infrastructure implementation details.
A couple of things worth noting about how these capabilities behave in practice:
- Airflow log tools (e.g.
get_failed_task_logs) don't dump raw text into the LLM's context window, since that would blow up token limits fast. Instead they parse logs for known signatures (exit code 137,OOM,AccessDenied) and return a normalized payload like{"contains_oom_kill": true, "exit_code": 137}.
- CloudWatch/CloudTrail tools (e.g.
get_lambda_errors,get_iam_policy_changes) use standardboto3clients, and infer the relevant resource (like a Lambda function name) directly from the incident's DAG name, returning a strictly typed error count rather than a raw metrics blob.
- Infrastructure tools (e.g.
get_redis_queue_depth) check physical infrastructure state directly, flagging something likecelery_queue_saturated=Truewhen Celery queue depth crosses a threshold.
Architect's Note: Evidence Before Tokens. Instead of sending raw infrastructure logs to the LLM, AirGuard preprocesses telemetry into structured operational signals first. A multi megabyte Airflow stack trace becomes{"contains_oom_kill": true, "exit_code": 137}. This significantly reduces token usage while giving the deterministic engine consistent, machine readable evidence instead of unbounded text.
This design pays off in three ways. It protects against prompt injection since the LLM can't run arbitrary code. It saves tokens. And it hands the deterministic engine perfectly structured evidence objects to reason over downstream.
Step 3: Deterministic Evidence Correlation
With evidence collected, the LLM steps aside. The
DeterministicInvestigationEngine normalizes raw data into boolean signals, such as task_oom_killed=True and aws_lambda_error_count_high=True , and runs them through a correlation engine of explicit rules against a pre loaded taxonomy of failure signatures. That taxonomy spans six operational pillars, covering areas like compute and infrastructure health, cost, security and access, scheduling and retries, data, and ML training, each holding a cluster of related signatures such as Worker OOM, Retry Storm, or SageMaker Cost Spike.RULES = [
PhantomRetrainingStormRule(),
SageMakerTimeoutLoopRule(),
UpstreamCascadeRule(),
]
def run(graph, bundle):
findings = []
for rule in RULES:
findings.extend(rule.evaluate(graph, bundle))
return findings
class PhantomRetrainingStormRule(CorrelationRule):
def evaluate(self, graph, bundle):
signals = bundle.signals
if not signals.cost_delta_high:
return []
if signals.backfill_count < BACKFILL_STORM_THRESHOLD:
return []
return [
CorrelatedFinding(
finding="Cost anomaly strongly correlated with excessive Airflow backfill runs.",
related_evidence=graph.select(
domains=[
"aws_cost_explorer",
"airflow_history"
]
),
severity="high"
)
]Step 4: Synthesizing the Root Cause
Instead of asking the LLM to guess at a root cause, the engine sorts
CorrelatedFinding objects by severity and relevance and explicitly constructs the narrative, guaranteeing every root cause is backed by specific telemetry.Step 5: Confidence Scoring via Corroboration
How should an engineer know whether to trust a given report? One telemetry source rarely tells the complete story, so AirGuard scores confidence based on cross domain corroboration rather than model certainty. If Airflow logs say a task timed out, and CloudWatch shows a Lambda timeout spike in the same window, confidence goes up accordingly:
Architect's Note: Confidence Comes From Corroboration. Repeated DAG retries become far more convincing when CloudWatch simultaneously reports elevated Lambda timeouts and CloudTrail records an infrastructure change in the same execution window. Operational confidence reflects evidence corroboration across independent systems, not a probabilistic language model output.
Step 6: Closing the Loop in Slack
Once the engine has a root cause and a confidence score, AirGuard doesn't dump a wall of text into a channel. It uses Slack's Block Kit to build a structured, interactive report: a header (e.g. "Automated Investigation: Worker OOM Crash"), status, an executive summary, and the root cause with its confidence score. Where the deterministic engine identifies a clear, low risk remediation, it also attaches interactive approval buttons, letting an engineer approve or reject the recommended fix directly from the message.
If AirGuard deduces that a failure was a transient API timeout, for instance, the recommendation is to clear the task state, and the Slack message ships with an "Approve" button. Clicking it sends a callback to AirGuard, which executes the Airflow API call directly, resolving the incident from the Slack channel without anyone opening a laptop.
Engineers appreciate a well written report, but expect the operational facts underneath it to stay immutable. So AirGuard completes timeline reconstruction, root cause analysis, confidence scoring, and recommendations before any language model touches the output. When enabled, Amazon Nova receives only the completed investigation report, with explicit instructions to improve readability without introducing new information.
Architect's Note: Deterministic First, LLM Last. Presentation stays separate from operational reasoning. The LLM's job at this stage is to polish prose, not to add or reinterpret facts. The executive summary is a rendering of the investigation's conclusions, never a source of new ones.
Design Decisions and Trade-offs
Several architectural choices shaped AirGuard, each made by weighing an alternative that looked easier in the short term against what it would cost in reproducibility, safety, or maintainability later:
Challenge | Alternatives Considered | Decision | Rationale |
|---|---|---|---|
Root cause generation | LLM inference vs. deterministic engine | Deterministic engine | Reproducibility and auditability |
Evidence collection | Static pipeline vs. dynamic planning | AgentCore | Reduces unnecessary API calls |
Integration | Airflow plugin vs. external platform | External event driven platform | Avoids modifying orchestration infrastructure |
Extensibility | Hardcoded integrations vs. capability registry | Capability registry | Easier addition of new services |
A Reference Architecture, Not Just a Tool
AirGuard is intended as a reference architecture for building trustworthy, agentic operational investigation systems. The current implementation focuses on Apache Airflow and AWS, but the underlying architecture is designed to support additional orchestration platforms, cloud providers, and operational domains through its capability driven investigation model.
Try It Yourself
Want to see an autonomous investigation in action? We've containerized the entire architecture so you can spin it up locally in minutes.
1)Set up your environment.
Clone the repository and copy the environment template. You can run the core deterministic engine with zero configuration, but adding your AWS credentials and a Slack Bot Token will unlock the LLM polish and interactive Block Kit reports.
git clone https://github.com/ja616/AirGuard.git
cd AirGuard
cp .env.local.example .env.local2)Spin up the infrastructure.
AirGuard requires a distributed environment to shine. The Docker Compose stack boots up Apache Airflow, PostgreSQL, Redis, and the AirGuard FastAPI backend in one shot.
docker compose up -d --buildWait about 45–60 seconds for the Airflow scheduler and webserver to fully initialize.
3)Launch the dashboard.
AirGuard ships with a Next.js frontend that visualizes the agent's reasoning and the deterministic engine's correlation in real time.
cd frontend && npm install && npm run dev
Open localhost:3000 in your browser to view the Command Center.
4)Trigger a Real Incident.
You don't need a simulation script. AirGuard is already wired into the local Airflow instance via native on_failure_callback webhooks. Open the Airflow UI at localhost:8080 (login: admin/admin) and manually trigger the daily_ml_pipeline DAG. Watch the pipeline run. When the train_sagemaker_model task inevitably fails, Airflow will automatically fire a webhook payload directly to the AirGuard backend.
5)Watch the Detective & Judge at Work.
Jump back to your AirGuard Command Center at localhost:3000. You'll instantly see the real-time investigation pipeline spin up. It transitions from Queued to Collecting Evidence (where the LLM dynamically queries the Airflow API) to Correlating (where the deterministic engine judges the evidence). Within seconds, your Slack channel will ping with a fully deterministic, telemetry-backed root cause analysis of the failure you just triggered.
(Note: For those in a hurry, you can also instantly fire a webhook from your terminal using python scripts/trigger_investigation.py).
Conclusion
Most operational tooling either watches a single system closely or gives an LLM free rein to guess. AirGuard takes a narrower position: let an agent decide where to look, and let a deterministic engine decide what it means. That boundary is what turns a plausible sounding explanation into one an engineer can actually act on, and what lets the same incident produce the same answer every time it's investigated.
The result isn't a smarter dashboard. It's a system that does the correlation work engineers currently do by hand, and hands back a root cause with the evidence to back it, before anyone's opened a laptop.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article