AWS Builder Center

Builder Center

From Silent Failures to Reliable Serverless Systems: Building an Observable AWS Pipeline

Explore how to troubleshoot a serverless document-processing pipeline on AWS using structured logging, CloudWatch metrics and alarms, retry strategies, dead-letter queues, and cost-aware monitoring.

Sreeja R

Introduction

Serverless applications can be easy to start and surprisingly difficult to operate reliably in production.
Imagine a document-processing application where users upload files to Amazon S3. An AWS Lambda function is triggered to process each upload, extract relevant information, and save the resulting metadata in Amazon DynamoDB.
Initially, everything works well. As usage grows, however, some documents remain unprocessed, certain requests take longer than expected, and users occasionally receive incomplete results. The Lambda function may even report successful execution while a downstream operation has failed.
The challenge is no longer simply writing a function. It is understanding what happened, identifying the failure quickly, recovering safely, and preventing the same problem from recurring.
This article explores a practical architecture for improving the observability and reliability of such a serverless pipeline.

1. Understanding the Architecture

A basic document-processing workflow can use the following AWS services:
  • Amazon S3: Stores uploaded documents and emits object-created events.
  • AWS Lambda: Processes documents in response to events.
  • Amazon DynamoDB: Stores document status, metadata, and processing results.
  • Amazon CloudWatch: Collects logs, metrics, and alarms.
  • Amazon SQS: Buffers work and can provide a dead-letter queue for failed messages.
For workloads requiring more control over retries, concurrency, and backpressure, S3 events can send messages through SQS before Lambda processes them.
The resulting flow is:
S3 upload → SQS queue → Lambda processing → DynamoDB update → CloudWatch monitoring.
This is a reference architecture rather than a claim that every application needs all five services. A simple workload may not require SQS, while a high-volume or failure-sensitive workload can benefit significantly from the additional queueing layer.

2. Scenario: When a Successful Invocation Hides a Failed Workflow

Consider a Lambda function that extracts information from an uploaded document and stores the result in DynamoDB.
A common implementation mistake is to catch an exception, log a message, and then return normally:
1
2
3
4
5
6
try:
save_document_metadata(document_id, metadata)
except Exception as exc:
logger.error("Metadata update failed: %s", exc)

return {"status": "success"}
This creates a misleading result. Lambda sees a normal return, even though the business operation failed.
For asynchronous invocation, a successful return generally tells the invocation system that processing completed successfully. If the application suppresses a genuine processing error, the normal failure and retry mechanisms may not activate as intended.
A better approach is to log the failure and re-raise it when the operation has not succeeded:
1
2
3
4
5
6
7
8
try:
save_document_metadata(document_id, metadata)
except Exception:
logger.exception(
"Document metadata update failed",
extra={"document_id": document_id}
)
raise
The exception should only be re-raised when the failure is genuinely unrecovered. If the application deliberately handles a known condition, that handling should be explicit and observable.
The key lesson is simple: a technical success response must represent actual completion of the intended operation.

3. Use Structured Logs to Find the Root Cause

Plain-text logs can be difficult to search when hundreds of documents are processed concurrently.
Structured logs make important fields easier to filter and correlate. Useful fields include:
  • request_id
  • document_id
  • processing_stage
  • duration_ms
  • status
  • error_type
For example, a log entry could look like this:
1
2
3
4
5
6
7
8
{
"level": "ERROR",
"event": "document_processing_failed",
"document_id": "doc-123",
"processing_stage": "metadata_persistence",
"error_type": "ThrottlingException",
"duration_ms": 1840
}
This is an illustrative log entry, not a record from a live AWS account.
With consistent fields, an engineer can search for a specific document, identify which stage failed, and compare failures across time periods.
Avoid logging document contents, credentials, access tokens, or unnecessary personal information. Observability should improve troubleshooting without exposing sensitive data.

4. Turn Logs into Actionable Monitoring

Collecting logs is not the same as detecting a problem.
Amazon CloudWatch supports metrics, dashboards, and alarms that help teams recognize operational issues before users report them.
Useful signals include:
Lambda errors: Detects failed invocations.
Lambda duration: Reveals slow processing and potential timeout risks.
SQS queue depth and message age: Helps identify growing backlogs and work that is not being processed quickly enough.
Dead-letter queue messages: Highlights messages that exhausted their configured retry path.
Business-level completion rate: Measures whether uploaded documents actually reach the completed state.
That last signal is particularly important. A Lambda function can have a low error count while the overall workflow still fails because of incorrect status handling, skipped steps, or downstream inconsistencies.
Create alarms around meaningful thresholds, choose notification destinations appropriate to the team, and test the alerting path. An alarm that nobody receives or acts upon provides little operational value.
CloudWatch metric names and dimensions vary by service and configuration, so select the actual metrics that match the architecture you deploy.

5. Design Retries and Dead-Letter Queues Carefully

Transient failures happen. A downstream service might throttle requests, a network call might time out, or a dependency might be temporarily unavailable.
Retries can help, but unlimited or poorly controlled retries can increase costs and make an incident worse.
A robust design should:
  1. Distinguish transient errors from permanent input or validation errors.
  2. Configure retry behavior appropriate to the invocation model.
  3. Use bounded retries and suitable backoff where the application controls retrying.
  4. Route messages that cannot be processed successfully to a dead-letter queue when supported by the chosen configuration.
  5. Alert when messages accumulate in the dead-letter queue.
  6. Provide a controlled process for inspecting and redriving failed messages.
An important distinction is that Lambda asynchronous invocation and Lambda event source mappings, such as SQS-triggered processing, have different retry and failure-handling behavior. Configure the mechanism that actually applies to your architecture rather than assuming every trigger behaves identically.
A dead-letter queue is not a permanent solution by itself. Someone must investigate the cause, correct the issue, and decide whether replaying the message is safe.

6. Make Processing Idempotent

Retries introduce another production concern: duplicate processing.
For example, a document may be processed successfully, but the acknowledgement or status update may fail. The same event could then be delivered again.
An idempotent operation produces the intended result even if the same logical request is processed multiple times.
Possible approaches include:
  • Assigning each document or processing job a stable identifier.
  • Recording processing status in DynamoDB.
  • Using conditional writes to control state transitions.
  • Making downstream updates safe to repeat.
  • Designing external side effects so that retries do not create duplicate business actions.
For workflows involving multiple steps, a state machine or explicit processing-status model can help distinguish pending, processing, completed, and failed work.
The exact strategy depends on whether duplicate execution is harmless, expensive, or capable of creating irreversible side effects.

7. Keep Observability Cost-Aware

Production monitoring should improve reliability without generating unnecessary expenses.
A few practical measures include:
  • Set appropriate CloudWatch log retention instead of retaining every log indefinitely.
  • Avoid verbose debug logging for every event when it is not needed.
  • Use targeted metric filters and alarms rather than creating large numbers of redundant resources.
  • Set AWS Budgets and review actual service usage.
  • Monitor queue depth and processing duration so that failures do not silently create growing workloads.
  • Review data transfer, storage, and invocation costs as traffic increases.
AWS Budgets can help notify you when spending reaches configured thresholds, but budget notifications are not the same as a hard spending cap.
Also, remember that creating Lambda functions, SQS queues, CloudWatch log groups, alarms, and related resources may incur charges depending on usage, region, and applicable free-tier benefits. Review current pricing before deploying a test environment.

8. A Practical Validation Checklist

Before considering the pipeline production-ready, test more than the happy path.
  • Upload a valid document and verify that processing completes.
  • Provide invalid input and verify that the failure is visible.
  • Simulate a transient dependency failure and confirm that the configured retry behavior works.
  • Test what happens after retries are exhausted.
  • Confirm that dead-letter messages generate an actionable alert.
  • Replay a message and verify that duplicate processing does not corrupt the result.
  • Check that dashboards reflect real processing outcomes.
  • Confirm that logs contain useful diagnostic information without exposing sensitive data.
  • Review log retention, alarms, queue configuration, IAM permissions, and expected cost.
These tests turn reliability from an assumption into something that can be demonstrated.

Conclusion

Reliable serverless architecture is not achieved simply by deploying managed services. It comes from making failures visible, ensuring that errors are handled honestly, designing safe retries, preventing duplicate side effects, and monitoring business outcomes as well as infrastructure metrics.
For a document-processing pipeline, combining structured logging, CloudWatch alarms, deliberate retry behavior, dead-letter queues, and idempotent processing creates a stronger foundation for production operations.
The most useful question is not just, “Did my Lambda function run?” It is, “Did the user's intended workflow finish correctly, and can I prove it?”
That shift in perspective is what makes a serverless application easier to operate and more trustworthy as it grows.
# aws-lambda# amazon-cloudwatch# amazon-s3# serverless
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article