Your GenAI Application Is Healthy. So Why Is It Giving Bad Answers?

Extending AWS observability beyond HTTP 200 to detect, investigate, and recover from silent failures in generative AI applications.

Mario Rodrigo Serrano Pineda

Published Oct 6, 2026

Last modified Oct 6, 2026

Spanish version: https://builder.aws.com/content/3KKGNUlrE4BXxvqtZC3bLcQJO8d/tu-aplicacion-genai-esta-saludable-entonces-por-que-esta-dando-malas-respuestas

From a Community Day talk to a deeper technical discussion

This article grew out of my session “When Your AI Fails at 3 AM: How to Detect, Investigate, and Recover from Silent Failures in GenAI Applications on AWS,” presented at the AWS Data & AI Community Day.

During the session, I explored a reliability problem that becomes increasingly important as generative AI applications move into production: an application can be technically healthy while still failing its users.

We built and demonstrated this scenario using a GenAI application on AWS, where Amazon API Gateway, AWS Lambda, Amazon Bedrock, a knowledge base, and Amazon CloudWatch continued operating normally—even while the application started producing low-quality answers.

The session focused on three questions:

How do we detect a failure when there is no HTTP 500?
How do we investigate what changed when the infrastructure looks healthy?
And how can SRE practices help us recover safely?

This article goes deeper into one of the key ideas behind that session: semantic observability—extending traditional observability so that we monitor not only whether an AI application responds, but also whether its responses remain useful, grounded, and aligned with what the system was designed to do.

If you want to see the implementation behind the concepts discussed here, you can also explore the hands-on demo and architecture in the GitHub repository, or watch the video walkthrough on YouTube.

▶ Watch the demo: https://youtu.be/j8LPjJyjKNw
⌨ Explore the implementation: https://github.com/marosepi2020/proyectos/tree/main/AWS%20Community%20Day%20-%20Data%26AI-Genai-Silent-Failure

----------

Imagine this scenario.

It is 3:00 AM.

Your support team reports that the company's AI assistant has started giving users incorrect, irrelevant, or incomplete answers.

You open your monitoring dashboard.

Everything looks healthy.

API requests are returning HTTP 200.

AWS Lambda functions are executing successfully.

Amazon Bedrock is responding.

Latency is within the expected range.

There are no infrastructure alarms.

From a traditional operational perspective, the system is healthy.

But from the user's perspective, the application is failing.

This is one of the challenges teams encounter when moving generative AI applications from experiments into production:

Technical availability does not necessarily mean AI quality.

For DevOps and SRE teams, this changes the way we need to think about observability.

The invisible failure

Traditional applications usually expose failures through signals that infrastructure and application monitoring systems understand well.

A request returns HTTP 500.

A Lambda invocation fails.

Latency increases.

CPU utilization reaches a threshold.

A downstream service becomes unavailable.

These signals fit naturally into the traditional observability model of metrics, logs, and traces.

Generative AI introduces another category of failure.

The infrastructure can work perfectly while the answer itself is wrong.

Consider a Retrieval-Augmented Generation (RAG) application running on AWS.

A typical request might follow a path such as:

User → Amazon API Gateway → AWS Lambda → Amazon Bedrock → Knowledge Base → Foundation Model → User

Every component may complete its operation successfully.

But the application could still return:

an answer unrelated to the user's question;
information unsupported by the retrieved context;
an incomplete answer;
outdated information;
an answer that violates the expected response format;
or a response that sounds convincing but is factually incorrect.

There may be no exception.

No HTTP 500.

No timeout.

No failed Lambda invocation.

From the infrastructure perspective:

SUCCESS

From the user's perspective:

FAILURE

That gap is where traditional observability becomes insufficient.

From system health to answer quality

When operating GenAI workloads, I find it useful to separate observability into two complementary dimensions.

1. Technical observability

This answers questions such as:

Is the API available?
Is Lambda executing correctly?
How long does the request take?
Is Amazon Bedrock responding?
Are there throttling errors?
How many tokens are being consumed?
How much does each interaction cost?

Typical signals include:

Latency, error rate, throughput, availability and resource consumption.

These signals remain essential.

We should not replace traditional observability.

But we need to extend it.

2. Semantic observability

Semantic observability asks a different question:

Was the answer actually good?

Instead of monitoring only whether the request succeeded, we start observing characteristics of the response itself.

Depending on the application, useful quality signals could include:

relevance;
groundedness;
completeness;
faithfulness to retrieved context;
response-format compliance;
safety;
or task success.

Now the operational picture becomes much more interesting.

Instead of seeing only:

HTTPStatus = 200

we might see something like:

SemanticQualityScore = 0.42

The request technically succeeded.

But the quality signal tells us something may be wrong.

A simple architecture on AWS

Let's consider a simplified RAG architecture.

A user sends a question to an application.

Amazon API Gateway receives the request and invokes an AWS Lambda function.

The function retrieves relevant information from a knowledge base and invokes a foundation model through Amazon Bedrock.

The generated response is returned to the user.

Alongside that normal request path, we can introduce a second process responsible for evaluating response quality.

Conceptually:

The evaluator receives information such as:

User question + retrieved context + generated answer

and evaluates whether the response satisfies the application's quality criteria.

The result can then become an operational metric.

For example:

SemanticQualityScore = 0.91

or:

SemanticQualityScore = 0.38

Once quality becomes measurable, it can become observable.

Turning AI quality into an operational signal

This is where the approach becomes especially useful for SRE teams.

Suppose our application produces a quality score between 0 and 1.

We establish an initial threshold:

SemanticQualityScore >= 0.70 → acceptable

SemanticQualityScore < 0.70 → degraded

The exact threshold is not universal. It must be calibrated against the application's use case, evaluation dataset and tolerance for incorrect answers.

But once we have a meaningful signal, we can publish it as a custom metric in Amazon CloudWatch.

Now we can observe something traditional monitoring could not tell us before:

the application is available, but answer quality is degrading.

A CloudWatch alarm could detect repeated low-quality responses.

That alarm could trigger an investigation or an automated recovery workflow.

This creates an interesting evolution of the traditional SRE model.

We are no longer monitoring only infrastructure reliability.

We are beginning to monitor AI behavior as part of system reliability.

But detection is only the beginning

Suppose an alarm tells us:

Semantic quality has dropped below the expected threshold.

The next question is the same one SRE teams have always asked:

Why?

For a GenAI application, several components may influence the answer.

The problem might be:

the prompt;
retrieval quality;
missing documents;
stale knowledge;
incorrect chunking;
model behavior;
configuration changes;
application logic;
or a dependency.

This is why traces and correlation become extremely important.

For every interaction, we want enough context to reconstruct what happened.

For example:

request_id

prompt_version

model_id

retrieved_documents

retrieval_score

latency

token_usage

semantic_quality_score

Imagine investigating an incident and discovering this pattern:

Prompt v1 → quality ~0.91

Prompt v2 → quality ~0.54

Infrastructure metrics remain stable.

Now we have something actionable.

A recent prompt change may have introduced a semantic regression.

Without semantic signals, that regression might remain invisible until enough users complain.

Recovery: applying SRE thinking to GenAI

Detection alone is not enough.

Reliable systems should also have a recovery strategy.

Imagine that our application keeps the previous known-good prompt configuration available.

We deploy:

prompt-v2

Quality begins degrading.

Our monitoring detects repeated scores below the accepted threshold.

Instead of continuing indefinitely with the degraded configuration, the system can initiate a controlled response.

For example:

Detect → Investigate → Decide → Recover → Validate

Recovery could mean:

reverting to a previous prompt version;
switching retrieval configuration;
falling back to a safer model or workflow;
disabling a problematic feature;
routing the request for human review;
or activating a degraded but more reliable experience.

The important idea is not to automate every decision blindly.

The goal is to create controlled recovery mechanisms with clear guardrails.

This is where established SRE principles become extremely valuable for generative AI.

A new type of SLI

Site Reliability Engineering commonly relies on Service Level Indicators.

We measure things such as:

Availability

Latency

Error rate

Throughput

GenAI applications introduce an interesting possibility:

semantic SLIs.

For example:

Good Answer Rate

could represent:

acceptable responses / total evaluated responses

A team might then monitor:

GoodAnswerRate = 96.4%

This does not necessarily mean organizations should immediately create formal semantic SLOs for every AI application.

Evaluation itself has uncertainty.

But it gives teams a much better operational signal than simply asking whether the API returned HTTP 200.

What should we monitor?

A production GenAI workload should ideally provide visibility across several layers.

Layer	Example signals
Infrastructure	availability, CPU, memory, concurrency
Application	latency, errors, throughput
Model	tokens, invocation latency, throttling
Retrieval	documents retrieved, relevance, retrieval scores
Semantic quality	relevance, groundedness, completeness
Business	task completion, user feedback, conversion

The important lesson is that these layers complement each other.

Semantic observability does not replace infrastructure observability.

It completes the picture.

The role of DevOps and SRE is expanding

GenAI does not eliminate the need for DevOps or SRE practices.

It makes them more important.

Infrastructure still needs to be reliable.

Deployments still need controls.

Changes still need traceability.

Systems still need observability.

Incidents still need investigation.

But now another question joins the traditional ones:

Is the AI behaving the way the system was designed to behave?

That question connects AI engineering, observability and reliability engineering.

And I believe it will become increasingly important as organizations move more GenAI applications into production.

A practical maturity path

Teams do not need to build an advanced AI observability platform on day one.

A practical progression could be:

This allows teams to increase operational maturity without trying to solve every GenAI reliability problem at once.

Final thought

If there is one idea I would take away from this article, it is this:

A successful API request does not guarantee a successful AI interaction.

For traditional systems, HTTP 200 often tells us that the technical operation completed successfully.

For generative AI applications, we need another layer of understanding.

We need to observe not only whether the system responded, but whether the response was useful, grounded and aligned with what the application was expected to do.

That is where semantic observability becomes part of reliability engineering.

And it leads to an interesting question for the future of SRE:

If AI behavior is part of the system, shouldn't AI quality also be part of what we monitor?

Now I’d love to hear from you. This article is just the beginning. The next step is to turn these concepts into a practical, hands-on lab on AWS. Which topic would you like us to build next? Vote in the AWS Builder Center poll and help shape the next lab: https://builder.aws.com/poll/3KKAWnQP9BnM8sb31gIec5iALq6_po

The winning topic will become a new technical article, YouTube video, and hands-on lab for the community.

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article