AWS Builder Center
The Lambda That Cried Wolf: Why I Rebuilt Our Pipeline on Step Functions After Three Days of Silent Failures

The Lambda That Cried Wolf: Why I Rebuilt Our Pipeline on Step Functions After Three Days of Silent Failures

By Sumit Jagtap — Team NxtTech What a broken welfare app taught me about serverless orchestration, silent failures, and the architectural decision that saved our hackathon.

The 3 AM Problem

On day two of the AWS First Commit hackathon, our demo broke. Not loudly — the worst kind of breaking. The frontend loaded. The upload worked. But the diagnosis spinner spun forever.
I opened the browser console. No errors. I opened the CloudWatch logs for the frontend. Nothing. I opened the API Gateway logs. Request in, response out, 200 OK. The pipeline said everything was fine.
But nothing was fine.
It took me two hours to find the truth: our  extract  Lambda had thrown a  KeyError  on a document with a non-standard date format. Because it had been invoked synchronously by the previous Lambda, the exception was logged but never surfaced. The previous Lambda returned, the frontend got a 200, and the whole pipeline silently died two steps downstream.
I stared at the screen for a long minute and said out loud: "We need Step Functions."
That decision — to throw away 400 lines of chained Lambda calls and rebuild on AWS Step Functions — is the single most important architectural choice we made. This is the story of why.



The Naive Architecture (And Why It Failed)

Our first design was what every serverless beginner builds. One Lambda invokes the next:
# extract/handler.py (the wrong way)
def lambda_handler(event, context):
extracted = extract_fields(event)
match_response = lambda_client.invoke(
FunctionName="match",
Payload=json.dumps(extracted)
)
return json.loads(match_response["Payload"].read())
On paper, it's clean. Each Lambda does one thing. The chain is easy to read.
In production, it fails in ways you don't see:
1. Errors cascade silently. If  match  fails,  extract  doesn't know. If  extract  fails, the frontend gets a 500 with no context. There's no single place to see where the pipeline broke.
2. Retry logic becomes boilerplate. Every Lambda needs its own  try/except  block. Every retry decision is hand-coded. Every error path is bespoke.
3. The pipeline is invisible. There's no execution history. No visual timeline. No way to see what happened two weeks ago on case  abc-123  without grepping CloudWatch.
4. Cold starts compound. Each Lambda-to-Lambda call adds latency. Our four-step pipeline took 8–12 seconds end-to-end. Users on slow connections abandoned before seeing results.
We knew it was broken. We didn't know how broken until we started debugging a real failure.



The Decision That Changed Everything

I proposed switching to Step Functions. Our team had a hard 30-hour budget. A rewrite would cost 4 hours. It was risky.
We did it anyway. Here's what we gained in those 4 hours:

Built-in retry with exponential backoff

"Extract": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:...:function:ExtractFunction",
"Retry": [
{
"ErrorEquals": ["States.TaskFailed", "Textract.ThrottlingException"],
"IntervalSeconds": 2,
"MaxAttempts": 3,
"BackoffRate": 2.0
}
],
"Next": "Match"
}
That block replaces roughly 40 lines of Python we would have written in the extract Lambda. Multiply that by five stages. We deleted an estimated 200 lines of retry code and replaced them with declarative JSON.

Visual execution history

The Step Functions console shows every execution as a graph. Green nodes are successes. Red nodes are failures. You click a node and see the exact input and output.
I cannot overstate how much this changed our debugging velocity. A failure that used to take 20 minutes of log spelunking now takes 30 seconds.

Graceful fallbacks as first-class primitives

When our Bedrock integration failed on our AISPL account (the account-level restriction that would consume three days of our lives — see the main team blog), we needed the pipeline to keep running. Not to crash. Not to skip the diagnosis. To produce something.
With Step Functions, we added a  Catch  block:
"Catch": [
{
"ErrorEquals": ["States.ALL"],
"ResultPath": "$.error",
"Next": "GenerateFallback"
}
]
One block. The pipeline detects the failure, routes to a fallback state that generates the diagnosis from a template, and continues. Users get a result in every scenario. That's graceful degradation, and in a hackathon it's the difference between a working demo and a broken one.



The State Machine That Won Us the Demo

Here's our actual state machine. I'm including it in full because I think it's a useful reference for anyone building document-processing pipelines.
┌─────────────────┐
│ Extract │ Textract reads the uploaded document
│ (Lambda) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Match │ Fuzzy compare across documents
│ (Lambda) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ HasMismatches? │ Choice state
│ (Choice) │
└────┬────────┬───┘
│ │
Yes│ │No
▼ ▼
┌─────────┐ ┌─────────────────┐
│Generate │ │ GenerateFallback│
│(Bedrock)│ │ (Template) │
└────┬────┘ └────────┬────────┘
│ │
└────────┬───────┘
▼
┌─────────────────┐
│ UpdateStatus │ DynamoDB + SNS
│ (Lambda) │
└─────────────────┘
Each state passes the full execution context forward. Downstream Lambdas have everything they need without extra DynamoDB reads:
{
"caseId": "abc-123",
"s3Key": "uploads/rejection.txt",
"lang": "en",
"extractResult": {
"applicantName": "Ram Patil",
"fatherName": "Shyam Patil",
"scheme": "PM-KISAN"
},
"matchResult": {
"mismatchCount": 2,
"mismatches": [...]
},
"generateResult": {
"diagnosis": "...",
"correctionForm": "..."
}
}
Data flows through the state machine. There is one source of truth. There is no ambiguity about what state the pipeline is in.



The Fuzzy Matching Insight Nobody Told Me

While Step Functions solved our orchestration problem, the  match  Lambda solved the intelligence problem. This is the second-half of the story, and it's the part I'm proudest of.
We had to determine if two documents describe the same person. In Indian welfare documents, this is surprisingly hard:
Document A
Document B
Match?
Suraj Khanase
Suraj Kanase
Yes — spelling variant
Suraj Balasaheb Khanase
Suraj Khanase
Yes — middle name omitted
Ram Patil
Ram Patil (father: Shyam)
Yes — trailing metadata
Ram Patil
Ram Kumar
No
Shyam Patil
Shyam Patil
Yes
A naive  ==  comparison fails on the first three. We needed something smarter.
The insight: normalize hard, then measure character-level distance.
from fuzzywuzzy import fuzz

def normalize(name):
name = name.lower().strip()
name = re.sub(r'[^\w\s]', '', name) # remove punctuation
name = re.sub(r'\b(mr|mrs|smt|shri|dr)\.?\b', '', name) # honorifics
name = re.sub(r'\s+', ' ', name) # collapse whitespace
return name.strip()

def compare(name_a, name_b):
score = fuzz.ratio(normalize(name_a), normalize(name_b))
return {
"score": score,
"is_match": score >= 95,
"severity": "high" if score < 80 else "medium" if score < 95 else "low"
}
After normalizing,  Suraj Khanase  and  Suraj Kanase  differ by one character out of fourteen. That's 93% similarity — above threshold. We flag it as a spelling variant.
In our demo case,  Ram Patil  versus  Ram Name  scores 67%. Below 80. That's high severity — a real mismatch, not a variant.
Why not ML? Because fuzzy matching with hard normalization is:
  • Fast — milliseconds on Lambda, no cold model loading
  • Deterministic — same input, same output, every time
  • Debuggable — you can print the normalized strings and see exactly why the score is what it is
  • Tunable — the 95% threshold is a business decision, not a model hyperparameter
For a hackathon with four days, ML was never on the table. But even for production, I would start with fuzzy matching and only reach for ML when the false-positive rate becomes a real problem. Which it hasn't.



The Deployment Story: SAM and the Circular Dependency Trap

Writing the state machine was half the work. Deploying it was the other half.
We used AWS SAM for the entire stack. One command, seven Lambdas, API Gateway, DynamoDB, S3, Step Functions, SNS, EventBridge, all deployed together.
sam build && sam deploy --guided
First deploy: 4 minutes. Subsequent deploys: under 2 minutes. That speed — the ability to go from "I have an idea" to "it's live" in 90 seconds — is what makes SAM genuinely transformative for hackathons.
But we hit one trap that cost us 90 minutes.
The circular dependency problem.
Our state machine referenced Lambdas by ARN. Those Lambdas had an environment variable pointing back to the state machine's ARN (for triggering follow-up workflows). CloudFormation rejected the template:
Circular dependency between resources:
[AdhikarWorkflow, GenerateFunction, UpdateStatusFunction, ...]
The fix is subtle but generalizable: move shared environment variables out of  Globals . We had this:
Globals:
Function:
Environment:
Variables:
STATE_MACHINE_ARN: !Ref AdhikarWorkflow # ← creates the cycle
Every Lambda inherited this variable, including the Lambdas that the state machine referenced. Cycle.
We moved the variable to only the one Lambda that needed it (the  analyze  function that triggers the workflow):
AnalyzeFunction:
Type: AWS::Serverless::Function
Properties:
Environment:
Variables:
STATE_MACHINE_ARN: !Ref AdhikarWorkflow
The cycle broke. The deploy succeeded. And I learned a rule I've used in every SAM project since:  Globals  is convenient until it isn't. When you see a circular dependency, suspect  Globals  first.



What I Would Do Differently

Honest reflection.
1. Start with Step Functions on day one. We spent a day on the naive chained-Lambda approach. That was a day we could have spent on the frontend, or on better Textract parsing, or on testing with real documents. If your pipeline has more than two stages and any of them can fail, Step Functions is not premature optimization. It's the right starting point.
2. Use Express workflows for the hot path. We started with Standard workflows (full execution history, up to one year retention). Standard is expensive for high-volume short-duration flows. Express is 25x cheaper and faster for sub-5-minute pipelines. We migrated our demo path to Express on day three and should have done it on day one.
3. Log structured data, not strings. Every Lambda should log JSON, not formatted strings. When you're debugging across five Lambda functions and a state machine, you need grep-able, queryable logs. CloudWatch Insights becomes a superpower when your logs are structured.
4. Test with real documents before building the fallback. We built the fallback before we knew what real failures looked like. Half of the fallback path was wasted work. Real documents teach you what fails.



What I Learned

The architecture is the product. A working demo with the wrong architecture is fragile. A broken demo with the right architecture is fixable. On day two, our demo was broken but the architecture was recoverable. We rebuilt the pipeline in four hours and shipped. That was possible because we had the right primitives.
Serverless orchestration is not optional at scale. For a two-step pipeline, chain Lambdas. For a five-step pipeline with retries, fallbacks, and tracking, Step Functions is the only sane choice. Trying to build it with chained Lambdas is like writing your own HTTP library — technically possible, structurally wrong.
Fuzzy matching is underrated. Everyone reaches for ML. Sometimes the answer is one regex and one Levenshtein distance. The elegance of the fuzzy matching solution is that anyone on the team can read it, understand it, and tune it. That matters more than accuracy at the margin.
Graceful fallbacks are features, not compromises. Our Bedrock integration failed. Our fallback saved us. If we had designed for "Bedrock always works," we would have shipped nothing. If you design for "the primary path works and the fallback works," you ship every time.



Try It

The Step Functions definition is at  backend/statemachine/workflow.asl.json . The fuzzy matching logic is at  backend/src/match/handler.py . Both are MIT-licensed — steal them, improve them, tell me what you build.


Team NxtTech — Suraj Khanase (Lead), Sumit Jagtap, Nandkishor Tamkhade, Tejas Shinde
Built at AWS First Commit — Bharat Builds Tour, Event 01, September 2026.
If you're building serverless pipelines for Indian language documents, or you've hit the AISPL Bedrock restriction on a new account, I'd love to hear how you worked around it. Reach me through the GitHub repo.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article