AWS Builder Center
Diagnosing silent SQS → Lambda failures with AWS DevOps Agent

Diagnosing silent SQS → Lambda failures with AWS DevOps Agent

Most SQS → Lambda failures that break production never trigger the Errors metric. This article walks through four common failure modes, what their telemetry actually looks like, and what AWS DevOps Agent catches (and misses) when pointed at each one.

Getting the most out of AWS DevOps Agent on SQS and Lambda workloads

SQS to Lambda is one of the most common serverless integrations in production, and one of the most common sources of incidents that do not show up as a spike in the Lambda Errors metric. Several failure modes in this pattern are silent by default — no error metrics, no obvious log patterns, no alarms. Diagnosing them by hand requires correlating CloudWatch Logs, CloudWatch Metrics, CloudTrail events and IAM state, which is exactly the kind of work that AWS DevOps Agent is designed for.
This article is a practical guide for teams already running (or about to run) SQS → Lambda in production and who want to make the agent as effective as possible on this workload. It covers what to monitor, how to configure the agent space, and how to write investigation prompts that get useful results. The recommendations come from a small evaluation where AWS DevOps Agent was run against a fault-injection testbed for a consumer Lambda reading from SQS.
The short version: the agent is a good fit for this class of problem, but the quality of the investigation depends heavily on the setup.

The failure modes to watch for

Four failure modes drove this evaluation. The first one is the well-behaved case: its symptoms show up in standard Lambda metrics and in the DLQ. The other three are subtler — their symptoms show up in SQS-level metrics and CloudTrail rather than in Lambda Errors, which means teams that alert only on Lambda errors may take longer to notice them.
  • Poison pill → DLQ. A message that cannot be parsed, gets retried three times, lands in the DLQ. Visible in logs, in the Errors metric, and in DLQ depth. The easy case.
  • Silent message loss through a misconfigured event source mapping. When  ReportBatchItemFailures  is disabled but the handler returns  {"batchItemFailures": [...]} , SQS ignores the structured response and deletes every message in the batch on first delivery, including the ones the handler reported as failed. The Errors metric does not increment. The DLQ does not grow. The only trace is a single log line in CloudWatch.
  • Reserved concurrency set to zero. Every invocation is throttled before execution. No code runs, no logs are written, the Errors metric does not fire. Throttles spikes and Invocations stays at zero, which is the only signal.
  • Reprocessing loop from a denied  sqs:DeleteMessage  permission. The handler runs, writes to the downstream store successfully, and returns. But  DeleteMessage  is called by the Lambda service infrastructure outside the handler, and if the execution role is denied that action, the message is never removed from the queue. It is redelivered and processed again and again, with the same  MessageId  appearing repeatedly in the logs and the downstream store written each time.
Two of these four never increment the Lambda Errors metric. Standard error-based alerting misses them completely.

What to monitor

Lambda errors and throttles are still important signals — they catch the poison-pill case, application bugs, and any hard failure the handler raises on. On top of those, some queue-level metrics help spot the failures that don't always show up as Lambda errors:
  •  ApproximateNumberOfMessagesVisible  on the DLQ. Goes from 0 to non-zero when any message exhausts its retries. Catches the poison-pill case and the reprocessing loop after the redrive happens.
  •  ApproximateAgeOfOldestMessage  on the main queue. Catches backlog scenarios including throttling, consumer failures, and the reprocessing loop (which pins one message in flight indefinitely).
  • Ratio of  NumberOfMessagesReceived  to  NumberOfMessagesDeleted  on the main queue. When these diverge, something is wrong. The reprocessing loop shows as many receives and zero deletes. The throttled-concurrency case shows receives with no deletes either.
  •  Throttles  on the Lambda. Specifically useful for the reserved-concurrency case, where this is the only signal.
  •  Invocations = 0  while messages are arriving. Less common as an alarm pattern but catches the throttling scenario cleanly.
Alarms on Errors are still useful, but treated as one signal among several rather than the primary one.
Also worth knowing: silent message loss through the  ReportBatchItemFailures  misconfiguration is the one failure mode that cannot be caught by metric alarms alone. It requires either a check on event source mapping configuration at deploy time, or a CloudTrail-based alarm on  UpdateEventSourceMapping  events against production functions.

Setting up the DevOps Agent space

The agent's effectiveness depends on what it can see. Two setup choices make a measurable difference for SQS → Lambda workloads.

If you deploy with SAM

The default  AIOpsAssistantPolicy  grants broad read access, but not all of it is what you need when the deployed artifact lives in an S3 bucket. In particular:
  •  lambda:GetFunctionConfiguration  is included, but  lambda:GetFunction  is not.  GetFunction  is the action that returns a signed URL to the deployed  .zip .
  • Broad S3 metadata reads are included, but  s3:GetObject  on the SAM-managed artifact bucket (typically named  aws-sam-cli-managed-default-samclisourcebucket-* ) is not.
Without these two actions, the agent can see that a deployment happened and read function configuration, but cannot inspect the code itself. When the investigation calls for understanding what the code actually does, the agent has to rely on metadata and log shape — and in the evaluation this sometimes produced concrete but unsupported hypotheses about code behaviour.
Granting these two actions on your SAM artifact bucket and on the functions you want investigated is a small change with a large effect on the quality of the root cause analysis.

If your code is in Git

Connecting the source repository to the agent space gives the agent a second way to inspect the code, independent of the deployed artifact. It also lets the agent read anything else in the repo —  README.md , commit messages, architecture docs.
That last point matters in practice. A  README.md  that describes the workload clearly — what the service does, what its key dependencies are, any feature flags or conditional code paths, and links to runbooks or ADRs — gives the agent the same context a senior engineer on the team would have. When the agent sees unusual runtime behaviour, it can ground its reasoning in how the system is actually supposed to work rather than inferring purely from log shape. For a service with non-obvious conditional logic (multi-tenant branches, regional overrides, experimental code paths behind a flag), this is the difference between an accurate diagnosis and a plausible-but-wrong one.
Two things to keep in mind:
  • The most consistent way to benefit from this is to deploy through CI/CD (GitHub Actions, GitLab CI, CodePipeline) so that each Lambda deployment maps to a specific commit. When the runtime artifact and  main  are guaranteed to match, the agent can trust what it reads. Manual deploys from a developer machine break this guarantee and can lead the agent to hypothesise a divergence that does not actually exist.
  • A  README.md  that a new team member could read to understand the service in fifteen minutes is roughly the right level of detail for the agent too.

Other permissions worth checking

The default agent policy covers most of what a SQS → Lambda investigation needs —  cloudtrail:LookupEvents ,  lambda:ListEventSourceMappings ,  lambda:GetEventSourceMapping ,  iam:GetRole ,  iam:GetRolePolicy ,  iam:ListRolePolicies ,  sqs:GetQueueAttributes  are all included. Worth confirming rather than assuming, especially in environments with custom agent roles.
One detail that occasionally catches people out: IAM events are emitted to  us-east-1  regardless of the workload's region, since IAM is a global service. The agent has the permission to query CloudTrail in any region, but in the evaluation it did not always look in  us-east-1  automatically when investigating a workload running elsewhere. This is not a permission problem, but it's worth being aware of as a blind spot — more on this in the prompting section.

Writing investigation prompts that work

The evaluation used deliberately minimal prompts — the Lambda function name and a time window, nothing else — and the agent still reached correct diagnoses in most cases. Minimal prompts work.
That said, more information in the prompt leads to more focused investigations, and some specific hints help when the failure mode is one of the silent ones.
Things worth including when you have them:
  • The service or queue name, not just the Lambda name. The agent will investigate the consumer first, but naming the queue upfront saves a discovery step.
  • A narrower time window when you have one. "Over the last hour" is fine. "Between 14:00 and 14:30 UTC" is better if you know where the incident sits.
  • The symptom you're seeing. "Messages are not draining from the queue" or "DLQ depth grew unexpectedly" points the agent at the relevant failure class. Even vague symptom descriptions help.
  • A hint about IAM. If the investigation may involve IAM changes, mentioning that IAM events are emitted globally to  us-east-1  saves the agent a cycle. A sentence like "Check CloudTrail in  us-east-1  for IAM events in the incident window" is enough.
  • A pointer to the repo or relevant code paths if the workload has non-obvious conditional logic (feature flags, fault-injection hooks, tenant-specific branches).
Things not to put in the prompt:
  • A hypothesis framed as fact. "The Lambda role lost  DeleteMessage  permission" forces the agent to confirm or deny a specific theory rather than investigate from scratch. Better to describe the symptom and let the agent correlate.

Summary

For teams running SQS → Lambda in production who want DevOps Agent investigations to be effective:
  • Monitor queue behaviour, not just Lambda errors. DLQ depth, message age, and the receive-vs-delete ratio catch failure modes that never increment the Errors metric.
  • If you deploy with SAM, grant  lambda:GetFunction  and  s3:GetObject  on the artifact bucket to the agent role. Those two actions are what separate metadata-level investigation from code-level investigation.
  • Connect the source repository to the agent space, ideally with CI/CD-backed deployments so the runtime matches  main . Include a short  README.md  describing the workload.
  • Alarm on  UpdateEventSourceMapping  in addition to metric alarms, since silent misconfigurations of event source mappings cannot be caught by metrics alone.
  • Add specific information in the prompts (time window, symptom, IAM-in- us-east-1 ) when they're relevant. Minimal works; specific works better.
The agent does most of the work when given the right inputs, and is transparent about what it cannot reach. For SQS → Lambda workloads specifically, the setup above makes the difference between an investigation that stalls on metadata and one that reaches root cause in minutes.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article