AWS Builder Center

Troubleshooting Lambda Durable Functions with AWS DevOps Agent

Learn how to use AWS DevOps Agent to autonomously investigate failures in Lambda durable function workflows. This article demonstrates how DevOps Agent correlates CloudTrail events, CloudWatch logs, metrics, and X-Ray traces to identify root causes that span multiple functions and callback suspension boundaries. Includes real investigation scenarios and effective prompting strategies.

Technical Account Manager

Introduction:

Lambda durable functions let you build multi-step workflows that suspend and resume without incurring compute charges during wait periods. When a workflow depends on an external system (a payment gateway, an approval service, a third-party webhook), it suspends at a callback and resumes when the response arrives.
This pattern is powerful, but when something breaks, the root cause often spans multiple functionalities and a suspension boundary. A configuration change to one functionality can silently cause a different function to time out minutes later. Tracing that chain manually means querying CloudWatch Logs across multiple log groups, checking metrics for anomalies, inspecting X-Ray traces for gaps, and reviewing CloudTrail for changes.
AWS DevOps Agent automates this correlation. You describe the problem in natural language, and it investigates across all your observability data to produce a complete root cause analysis. For Lambda durable function workflows specifically, it excels at tracing failure cascades that cross function boundaries and the callback suspension gap.

How durable functions work with callbacks:

A typical Lambda durable function workflow with external callbacks involves multiple Lambda functions collaborating:
  • A function that starts the workflow and suspends at a callback, waiting for an external response
  • A separate function that receives the external event (e.g., a webhook) and calls  send_durable_execution_callback_success  to resume the suspended execution
For example, in a payment processing workflow: a client sends a payment request through API Gateway to a durable function. The durable function validates the request, creates a callback, calls the payment gateway's API to initiate the charge, and then suspends. When the payment gateway confirms the outcome via webhook, a separate Lambda function receives that webhook through API Gateway, verifies the signature, and calls  send_durable_execution_callback_success  to resume the original durable execution with the payment result.
The troubleshooting challenge lives in the suspension gap. The Lambda durable function is waiting. An external system must respond. A separate Lambda function must process that response and deliver the callback. Any failure in that chain results in the same symptom: the durable execution times out or fails. DevOps Agent traces backward from that symptom to find the actual cause.

Setting up DevOps Agent:

DevOps Agent connects to your account through an Agent Space:
  1. Open the DevOps Agent console in your region
  1. Create an Agent Space and associate your AWS account
  1. The agent automatically discovers Lambda functions, API Gateway, CloudWatch (logs, metrics, alarms), X-Ray traces, and CloudTrail events
No additional instrumentation is required beyond standard observability practices. The richer your existing signals, the more precise the investigation.

Investigation: callback timeout from a configuration change

A DurableExecutionFailures alarm fires. You ask DevOps Agent:
"Investigate recent failures in the payment processor function."
What DevOps Agent finds:
It traces the failure to a manual Lambda configuration change on the callback handler. Here is the cascade:
  1. Root cause — An  UpdateFunctionConfiguration  API call modified the callback handler's environment variables, breaking webhook signature verification. DevOps Agent pulls this from CloudTrail, including the IAM principal, timestamp, and source IP.
  1. Immediate effect — Webhook signature verification started failing 3 seconds after the config change. The agent finds the metric spike and correlating WARNING-level log entries.
  1. Downstream impact — Rejected webhooks meant the durable function's callback never received a response. After the configured timeout, the execution failed.
  1. Secondary issue — The external system retried the failed webhooks. After the callback had already expired, these retries caused  CallbackTimeoutException  errors and HTTP 502 responses.
  1. Recovery — A second configuration change restored the correct value. Processing resumed.
The timeline DevOps Agent produces:
11:23:20 UTC Config change (ROOT CAUSE)
11:23:23 UTC Webhook signature failures begin
11:24:17 UTC Signature failure alarm triggers
11:24:40 UTC DurableExecutionFailures alarm triggers
11:25:33 UTC Config restored
11:25:37 UTC Processing resumes
11:28:21 UTC Callback timeout (webhook never arrived during broken window)
Without DevOps Agent, this investigation requires manually correlating CloudTrail, multiple CloudWatch log groups, custom metrics, and alarm history. DevOps Agent does it in a single pass and also catches the secondary retry issue.

Investigation: deterministic errors with unnecessary retries

Multiple durable executions fail in a step that calls an external API. You ask:
"Why are durable executions failing at the create-payment-intent step?"
DevOps Agent finds that declined cards cause the step to retry 6 times over about a minute before failing. It identifies this as wasteful: a card decline is deterministic and will never succeed on retry. It recommends configuring the step to fail immediately on deterministic errors.
This insight comes from pattern analysis across multiple executions, not a single incident.

Investigation: external system not delivering callbacks

A batch of executions all time out within the same hour. You ask:
"What's causing the spike in callback timeouts?"
DevOps Agent investigates and finds:
  • All executions successfully called the external API
  • Zero callback responses were received during that window
  • No signature failures (webhooks never arrived at all)
  • No configuration changes in CloudTrail
It concludes the issue is external to your AWS environment: the third-party system is not delivering webhooks to your endpoint. It recommends verifying the endpoint configuration on the external system's dashboard.

Best practices for prompting DevOps Agent:

Effective prompts for durable function investigations:
Start broad, then narrow.
  • "Investigate recent failures in [function-name]" — lets the agent explore all signal types
  • Follow up with "What caused the callback timeout at [timestamp]?" to drill into a specific incident
Reference the symptom, not your hypothesis.
  • Good: "Why are durable executions timing out?"
  • Less effective: "Is the webhook secret wrong?" (this skips the investigation and may miss the actual cause)
Ask about correlations explicitly.
  • "Are there any configuration changes that correlate with the failures?"
  • "What changed between [time when it worked] and [time when it broke]?"
Use alarm names as entry points.
  • "Investigate the DurableExecutionFailures alarm that triggered at 11:24 UTC"
  • This gives the agent a precise starting point and time window
Ask for recommendations after root cause.
  • "What should I do to prevent this from happening again?"
  • The agent provides code-level, operational, and architectural recommendations
Include function names when you know the scope.
  • "Check the webhook handler and payment processor for errors in the last hour"
  • Naming specific functions helps the agent focus on relevant log groups

What to include for effective troubleshooting:

DevOps Agent is most effective for durable function troubleshooting when:
  • Correlation keys are present in logs. Shared identifiers (callback_id, request_id, transaction_id) across functions let the agent trace a single transaction through the entire workflow, including across the suspension boundary.
  • Metrics exist at each workflow stage. Stage-by-stage metrics let the agent pinpoint exactly where in the workflow the failure occurs, rather than just reporting that the execution failed.
  • X-Ray tracing spans the full request path. Active tracing on all functions and API Gateway gives the agent visibility into latency, errors, and gaps in the request flow.
  • CloudTrail is enabled. Configuration drift is a common root cause for callback failures. CloudTrail gives the agent the ability to correlate changes with failure onset.
  • Alarms define clear failure boundaries. Well-defined alarms (one per failure mode: execution failure, callback timeout, signature failure) give the agent precise entry points for investigation.
  • Logs include explicit callback lifecycle events. Log entries for callback creation, suspension, resumption, and timeout create clear markers that the agent uses to determine execution state at any point in time.
When these are in place, DevOps Agent can autonomously investigate a multi-function durable workflow failure and produce an accurate root cause analysis with actionable recommendations, typically in under a minute.

Conclusion:

Lambda durable functions with external callbacks create workflows where failures cascade across function boundaries and suspension gaps. The root cause of a callback timeout might be a configuration change, a signature verification failure, or an external system outage, and identifying which one requires correlating multiple signals across multiple services.
AWS DevOps Agent automates this. Point it at a failing durable function, and it traces the full cascade from trigger to symptom, identifies secondary issues, and provides actionable recommendations. Structure your logs with correlation keys, define alarms for each failure mode, and DevOps Agent handles the investigation.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article