When AI Knows It Doesn’t Know: Building Safer Robotic Systems with Confidence and Fallbacks

How confidence-aware AI and fallback strategies can make autonomous systems safer and more resilient

Ana Clara dos Santos

eng UFRJ

Published Oct 5, 2026

Last modified Oct 5, 2026

Introduction

Artificial intelligence is increasingly moving from purely digital environments into the physical world.

A recommendation system can make a bad recommendation. A chatbot can misunderstand a question. But when an AI system controls a robot, an incorrect prediction can become a physical action with real consequences.

Consider a mobile robot navigating a warehouse. Its camera detects an object in its path and a computer vision model classifies it as a box with 97% confidence. The robot slows down and changes direction.

Now imagine a different situation. The camera is partially obstructed, the lighting is poor, and the model classifies the object as a box with only 54% confidence.

Should the robot make the same decision?

A conventional AI pipeline might simply return the model's prediction and allow the application to continue. A more robust autonomous system should instead consider how reliable that prediction is before deciding what action to take.

This leads to a simple but important idea:

A reliable autonomous system is not one that is always confident. It is one that knows what to do when it isn't.

In this article, we will explore how to design a confidence-aware robotic architecture in which an AI model is only one part of the decision-making process.

The proposed system will use a combination of edge computing, machine learning inference, confidence thresholds, deterministic fallback behaviors, and cloud-based monitoring. AWS IoT Greengrass can perform local machine learning inference on edge devices, while Amazon SageMaker AI can be used for model training and hosted inference. AWS IoT Core provides MQTT-based communication between devices and the cloud.

The goal is not to make the robot "perfect".

The goal is to make it behave safely when the AI is uncertain.

1. The Problem with AI Predictions

Machine learning models are usually evaluated using metrics such as accuracy, precision, recall, F1 score, or mean average precision.

These metrics are extremely useful.

However, they don't completely answer the question we need to ask when deploying AI into an autonomous system:

Can the robot safely act on this particular prediction right now?

Suppose a classification model achieves 95% accuracy on a test dataset.

That does not mean that every prediction produced by the model is 95% reliable.

The model may encounter:

lighting conditions that were not present in the training data;
objects from an unfamiliar environment;
camera obstruction;
sensor noise;
unusual object orientations;
motion blur;
changes in weather;
hardware degradation;
or inputs that are simply outside the model's training distribution.

A model can therefore produce a prediction even when the input is very different from anything it learned during training.

This creates an important distinction:

Model accuracy describes aggregate performance across a dataset.

Decision confidence describes how strongly the model supports an individual prediction.

They are related, but they are not the same thing.

And in robotics, that distinction matters.

2. From Prediction to Decision

A common architecture looks like this:

1
2
3
4
5
6
7
Sensor
   ↓
AI Model
   ↓
Prediction
   ↓
Robot Action

The problem is the last transition.

The prediction is treated almost like a command.

A safer architecture inserts an additional layer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Sensor
   ↓
AI Model
   ↓
Prediction + Confidence
   ↓
Decision Layer
   ↓
┌───────────────┬────────────────┬─────────────────┐
│ High          │ Medium         │ Low             │
│ confidence    │ confidence     │ confidence      │
├───────────────┼────────────────┼─────────────────┤
│ Normal action │ Safe action    │ Fallback        │
│               │ / verification │ behavior        │
└───────────────┴────────────────┴─────────────────┘

This changes the role of the AI.

The model is no longer responsible for deciding everything.

Instead, it becomes a perception component that provides information to a separate decision layer.

This separation is particularly valuable in safety-sensitive systems because deterministic safety rules can remain active even when the machine learning model behaves unexpectedly.

3. A Simple Confidence Policy

For a prototype, we can define three confidence ranges.

Confidence	Interpretation	Robot behavior
≥ 0.80	High	Execute normal behavior
0.50–0.79	Uncertain	Slow down / verify
< 0.50	Low	Trigger fallback

These values are only examples.

In a production system, thresholds should not be chosen arbitrarily. They should be determined through validation experiments, operational requirements, risk analysis, and the characteristics of the specific model and environment.

The important idea is the policy, not the exact numbers.

For example:

1
2
3
4
5
6
7
8
def decide_action(confidence):
    if confidence >= 0.80:
        return "NORMAL_ACTION"

    if confidence >= 0.50:
        return "SAFE_MODE"

    return "FALLBACK"

Now imagine the model produces:

1
2
Prediction: obstacle
Confidence: 0.94

The system can safely select:

1
NORMAL_ACTION

But if the model produces:

1
2
Prediction: obstacle
Confidence: 0.57

the robot does not necessarily need to stop immediately.

It can switch to a conservative behavior:

1
2
3
4
SAFE_MODE
→ reduce speed
→ collect additional sensor information
→ attempt another inference

And with:

1
2
Prediction: obstacle
Confidence: 0.31

the system can choose:

1
2
3
4
FALLBACK
→ stop
→ maintain safe state
→ request additional information or human intervention

This is a small architectural change, but conceptually it is significant.

4. Designing the System

For the proposed architecture, we can divide the system into four layers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌──────────────────────────────────────────┐
│              CLOUD LAYER                 │
│                                          │
│  Model training │ Monitoring │ Analytics │
└──────────────────────▲───────────────────┘
                     │
                     │ MQTT / IoT
                     │
┌─────────────────────┴────────────────────┐
│               EDGE DEVICE                │
│                                          │
│ Sensors → AI Inference → Decision Engine │
│                         ↓                │
│                   Robot Control          │
└──────────────────────────────────────────┘

The key design decision is that time-sensitive decisions happen at the edge.

A robot should not necessarily need to send every camera frame to the cloud and wait for a response before deciding whether to stop.

AWS IoT Greengrass is designed for this type of architecture. It allows applications and machine learning inference to run locally on supported edge devices while still connecting those devices to AWS services.

This gives us two complementary environments:

Edge

Responsible for:

sensor processing;
local inference;
confidence evaluation;
immediate decisions;
fallback behavior.

Cloud

Responsible for:

model training;
model management;
fleet monitoring;
historical analysis;
centralized data storage;
updating models and software.

AWS IoT Greengrass supports ML components that can deploy models and inference code to edge devices. AWS documents model, runtime, and inference components as separate building blocks, which also makes it possible to customize the architecture rather than treating inference as a monolithic application.

5. Where Amazon SageMaker AI Fits

The machine learning model does not necessarily need to be trained on the robot.

A possible development pipeline is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Sensor Data
     ↓
Amazon S3
     ↓
Amazon SageMaker AI
     ↓
Model Training
     ↓
Model Evaluation
     ↓
Model Artifact
     ↓
AWS IoT Greengrass
     ↓
Robot

Amazon SageMaker AI provides managed capabilities for training and deploying machine learning models. It also supports real-time inference endpoints when inference needs to happen in the cloud.

For robotics, however, we should carefully decide which inference should happen where.

A cloud endpoint might be useful for:

complex models;
non-time-critical decisions;
centralized experimentation;
high-compute inference;
analysis and validation.

Local inference is preferable when:

latency is critical;
connectivity may be intermittent;
the robot needs to react immediately;
sending raw sensor data to the cloud is expensive or undesirable.

AWS IoT Greengrass specifically supports local inference using models trained in the cloud, allowing these two approaches to coexist.

6. Building the Decision Engine

To demonstrate the idea without requiring a physical robot, we can simulate the decision layer in Python.

First, let's define the three possible operational states:

1
2
3
4
5
6
7
8
9
10
11
HIGH_CONFIDENCE = 0.80
LOW_CONFIDENCE = 0.50

def decide_action(confidence):
    if confidence >= HIGH_CONFIDENCE:
        return "NORMAL"

    if confidence >= LOW_CONFIDENCE:
        return "SAFE_MODE"

    return "FALLBACK"

We can then simulate several model predictions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
predictions = [
    ("person", 0.96),
    ("person", 0.73),
    ("unknown", 0.41),
    ("box", 0.89),
]

for label, confidence in predictions:
    action = decide_action(confidence)

    print(
        f"Prediction={label} "
        f"Confidence={confidence:.2f} "
        f"Action={action}"
    )

The output would look conceptually like:

1
2
3
4
Prediction=person Confidence=0.96 Action=NORMAL
Prediction=person Confidence=0.73 Action=SAFE_MODE
Prediction=unknown Confidence=0.41 Action=FALLBACK
Prediction=box Confidence=0.89 Action=NORMAL

Notice something important.

The same predicted class does not necessarily produce the same behavior.

If the model says:

1
person, 0.96

the system can use its normal policy.

If it says:

1
person, 0.55

the decision layer can treat the situation differently.

The prediction alone is not the decision.

7. Making the Fallback More Intelligent

A fallback should not simply mean "turn the robot off."

Different systems require different fallback behaviors.

For a mobile robot, fallback could mean:

1
2
3
4
5
6
7
8
9
10
11
12
13
LOW CONFIDENCE
      ↓
Reduce speed
      ↓
Collect additional sensor data
      ↓
Run inference again
      ↓
Confidence improved?
   ↙           ↘
 YES            NO
 ↓              ↓
Continue       Stop

For a robotic arm, it could mean:

1
2
3
4
5
6
7
Uncertain object
      ↓
Stop movement
      ↓
Move to predefined safe position
      ↓
Request verification

For an autonomous drone:

1
2
3
4
5
6
7
Uncertain perception
      ↓
Leave aggressive flight mode
      ↓
Maintain altitude
      ↓
Switch to conservative navigation

This is why fallback strategies should be designed according to the physical consequences of failure.

A low-confidence prediction should not have one universal response.

8. Adding Sensor Quality to the Decision

There is another problem with relying only on model confidence.

Suppose a camera becomes dirty.

The model might still output:

1
person: 0.91

The confidence looks high.

But the sensor itself may be unreliable.

Therefore, a more sophisticated system should consider both:

model confidence
sensor quality

We can represent this concept with:

1
2
3
4
5
6
7
8
9
10
11
def evaluate_decision(model_confidence, sensor_quality):
    if sensor_quality < 0.50:
        return "FALLBACK"

    if model_confidence >= 0.80 and sensor_quality >= 0.80:
        return "NORMAL"

    if model_confidence >= 0.50 and sensor_quality >= 0.50:
        return "SAFE_MODE"

    return "FALLBACK"

Now the system is no longer asking only:

"How confident is the model?"

It is asking:

"How much should I trust the complete perception pipeline right now?"

This idea can be extended further using multiple sensors.

For example:

1
2
3
4
5
Camera ──────────┐
                 │
LiDAR ───────────┼──→ Sensor Fusion → AI → Decision
                 │
IMU ─────────────┘

If the camera is uncertain but LiDAR provides strong evidence, the system may be able to continue operating.

If multiple sensors disagree, however, the system can become more conservative.

9. Connecting the Robot to AWS IoT Core

The edge device can publish telemetry to AWS IoT Core using MQTT.

For example:

1
2
3
4
5
6
7
8
{
  "robot_id": "robot-01",
  "timestamp": "2026-10-05T12:30:10Z",
  "prediction": "person",
  "confidence": 0.57,
  "sensor_quality": 0.82,
  "action": "SAFE_MODE"
}

AWS IoT Core supports MQTT, a lightweight messaging protocol widely used for connected devices.

A topic structure could be:

1
2
3
robots/robot-01/telemetry
robots/robot-01/decisions
robots/robot-01/alerts

This creates an important separation.

The robot does not need the cloud to make every immediate decision.

Instead, it can make the decision locally and report what happened.

That gives the cloud visibility without putting the cloud in the critical path of every physical action.

10. Monitoring AI Confidence

Once confidence becomes part of the system's behavior, it also becomes valuable telemetry.

Imagine monitoring:

1
2
3
4
5
6
7
8
9
10
Average confidence
↓
0.94
0.92
0.91
0.87
0.81
0.74
0.63
0.58

A gradual decrease could indicate that something has changed.

Possible causes include:

environmental changes;
sensor degradation;
a new type of object;
changing lighting;
model drift;
changes in the operating environment.

This means that confidence can become an early warning signal.

Instead of discovering that a robot is failing only after a dangerous event, operators could detect that the system is increasingly uncertain.

Cloud monitoring can therefore complement the local safety mechanism.

AWS IoT Greengrass provides components for collecting and uploading logs, while its telemetry capabilities can communicate device health information through AWS IoT Core.

11. Confidence Is Not the Same as Probability of Being Correct

There is an important machine learning caveat here.

A value such as:

1
confidence = 0.95

should not automatically be interpreted as:

"There is a 95% probability that the model is correct."

Whether a model's confidence is well calibrated depends on the model and how its outputs are produced.

This matters because our entire decision architecture depends on thresholds.

If a model systematically produces overconfident predictions, a threshold such as 0.80 may provide a false sense of security.

Therefore, a production implementation should include calibration and validation.

Possible approaches include:

calibration datasets;
reliability diagrams;
temperature scaling;
threshold optimization;
validation under different environmental conditions;
monitoring confidence distributions after deployment.

The decision layer should therefore not blindly trust a number called "confidence."

It should trust a validated signal whose meaning has been established experimentally.

12. Testing the Architecture

A useful prototype should not only demonstrate the happy path.

We should intentionally test situations where the system is likely to become uncertain.

For example:

Scenario	Model confidence	Sensor quality	Expected behavior
Normal environment	0.96	0.94	Normal
Moderate uncertainty	0.67	0.91	Safe mode
Poor camera	0.91	0.32	Fallback
Unknown object	0.38	0.88	Fallback
Multiple uncertain sensors	0.52	0.47	Fallback
Clear perception	0.89	0.93	Normal

This type of testing is more informative than simply measuring the model's average accuracy.

We are testing the behavior of the entire system.

That distinction is crucial.

A machine learning model can have excellent benchmark performance while the autonomous system built around it still has poor failure handling.

13. Edge vs. Cloud: Why Not Just Send Everything to the Cloud?

A natural question is:

Why not simply send sensor data to Amazon SageMaker AI and let the cloud make the decision?

Sometimes that is exactly the right solution.

But robotics introduces constraints that conventional cloud applications may not have.

Imagine a robot that needs to stop because an obstacle suddenly appears.

The system might need to:

1
2
3
4
5
6
7
8
9
10
11
12
13
capture sensor data
      ↓
send data
      ↓
network transmission
      ↓
cloud inference
      ↓
return prediction
      ↓
decision
      ↓
motor command

Every additional step introduces latency and another potential failure point.

With edge inference, the critical path can instead look like:

1
2
3
4
5
6
7
Sensor
  ↓
Local inference
  ↓
Decision engine
  ↓
Motor controller

The cloud can still receive the resulting telemetry.

This is one of the strengths of edge computing: the cloud and the device do not have to compete for responsibility.

They can perform different jobs.

14. A Complete AWS-Oriented Architecture

Putting the components together, the architecture could look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
                         AWS CLOUD
┌─────────────────────────────────────────────────────┐
│                                                     │
│  Amazon SageMaker AI                                │
│  ┌────────────────────┐                             │
│  │ Model Training     │                             │
│  └─────────┬──────────┘                             │
│            ↓                                        │
│       Model Artifact                                │
│            ↓                                        │
│      Amazon S3                                      │
│                                                     │
│  AWS IoT Core ←────────────── Telemetry             │
│       ↓                                             │
│  Monitoring / Analytics                             │
│                                                     │
└───────────────┬─────────────────────────────────────┘
                │
                │ MQTT
                ↓
             EDGE DEVICE
┌─────────────────────────────────────────────────────┐
│                                                     │
│ Sensors                                              │
│    ↓                                                │
│ Local preprocessing                                  │
│    ↓                                                │
│ AWS IoT Greengrass                                   │
│    ↓                                                │
│ ML inference                                         │
│    ↓                                                │
│ Prediction + confidence                              │
│    ↓                                                │
│ Decision engine                                      │
│    ├── Normal behavior                               │
│    ├── Safe mode                                     │
│    └── Fallback                                      │
│                                                     │
└─────────────────────────────────────────────────────┘

AWS IoT Greengrass supports deploying ML inference components to edge devices, including models trained in the cloud and stored in Amazon S3.

The important architectural principle is that the safety-critical decision path remains local.

The cloud provides intelligence at scale, but the robot retains the ability to respond to uncertainty.

15. Extending the Idea to ROS 2

This architecture can also be mapped naturally to a ROS 2-based robotic system.

For example:

1
2
3
4
5
6
7
8
9
/camera/image
      ↓
/perception_node
      ↓
/confidence_node
      ↓
/decision_node
      ↓
/cmd_vel

The perception node could publish:

1
2
class = "person"
confidence = 0.73

The decision node would then determine whether the robot should:

1
2
3
NORMAL
SAFE_MODE
FALLBACK

This separation is particularly useful because it prevents the machine learning component from directly controlling the robot's behavior.

The AI system provides information.

The decision layer determines what that information means operationally.

This makes it easier to replace the model later without rewriting the safety logic.

For example, we could replace:

1
Model A

with:

1
Model B

while keeping:

1
2
3
Decision Policy
Fallback Logic
Safety Constraints

unchanged.

That is an important software engineering advantage.

16. What About Generative AI?

Large language models can make this architecture even more interesting, but they should not necessarily sit directly in the safety-critical control loop.

A possible extension is:

1
2
3
4
5
6
7
8
9
10
11
Robot
  ↓
Safety system detects uncertainty
  ↓
Fallback behavior
  ↓
Context sent to higher-level AI
  ↓
LLM analyzes situation
  ↓
Human/operator receives explanation

For example:

"The robot entered safe mode because the camera confidence dropped below the operational threshold for three consecutive observations."

A generative model could help transform telemetry into human-readable explanations or assist an operator in diagnosing problems.

But there is an important distinction:

Using an LLM to explain a decision is different from allowing an LLM to directly control a safety-critical actuator.

The latter requires substantially stronger guarantees.

A useful principle is therefore:

Generative AI can augment the decision system without replacing the safety system.

17. Failure Is a First-Class State

Traditional application design often focuses on successful execution:

1
Input → Processing → Output

Autonomous systems require a different mindset:

1
2
3
4
5
6
7
8
9
10
11
12
Input
  ↓
Processing
  ↓
Can I trust this result?
  ↓
┌───────────────┐
│               │
YES             NO
│               │
↓               ↓
Act          Degrade safely

This means failure is no longer an exceptional condition.

It becomes an explicit state of the architecture.

That is perhaps the most important lesson of this design.

A robot does not need to know everything.

It needs to know when it should stop pretending that it knows.

18. Limitations

The architecture described here is intentionally simplified.

A real production system would require considerably more work.

Some important challenges include:

Confidence calibration

Model confidence must be validated rather than assumed to represent true probability.

Threshold selection

The correct threshold depends on the application and the cost of different types of failure.

Sensor fusion

A single sensor may not provide enough information for reliable decisions.

Adversarial and unexpected inputs

Confidence-aware systems can still fail when presented with inputs designed to fool the model.

Hardware failures

A model cannot compensate for a completely failed sensor or actuator.

Safety certification

Depending on the application, formal safety requirements and regulatory standards may apply.

Human factors

If humans are involved in the fallback process, the system must clearly communicate why intervention is required.

Therefore, confidence-aware decision-making should not be interpreted as a complete safety solution.

It is an architectural pattern for handling uncertainty more deliberately.

19. The Broader Idea

The concept extends beyond robotics.

The same pattern can be useful anywhere AI outputs trigger consequential actions.

Consider:

Industrial inspection

1
2
High confidence → automatically classify
Low confidence → send to human inspector

Healthcare decision support

1
2
High confidence → provide recommendation
Uncertain → request additional information

Fraud detection

1
2
High confidence → automatically flag
Uncertain → additional verification

Autonomous vehicles

1
2
3
High confidence → normal driving policy
Uncertainty → conservative driving
Critical uncertainty → minimal-risk maneuver

In each case, the architecture follows the same principle:

1
2
3
4
5
6
7
AI prediction
      +
uncertainty
      ↓
decision policy
      ↓
appropriate action

The model is not the entire system.

It is one component inside a larger decision architecture.

Conclusion

Artificial intelligence is often described in terms of how accurately it can predict.

For autonomous systems, that is only part of the problem.

A robot operating in the physical world needs to deal with uncertainty, unexpected inputs, degraded sensors, changing environments, and failures.

That means a robust AI system should not simply answer:

"What do I think this is?"

It should also answer:

"How much should I trust this result, and what should I do if I don't?"

By separating perception from decision-making, introducing confidence-aware policies, and designing explicit fallback behaviors, we can build systems that degrade more gracefully when machine learning becomes uncertain.

AWS provides the infrastructure to implement this pattern across both cloud and edge environments. Amazon SageMaker AI can support model development and managed inference, while AWS IoT Greengrass can bring machine learning inference and application logic closer to physical devices. AWS IoT Core can connect those devices to cloud services using MQTT.

But the most important part of the architecture is not any individual AWS service.

It is the design principle behind them:

Autonomy should not mean acting regardless of uncertainty. It should mean knowing how to respond to uncertainty.

And perhaps that is one of the next steps toward making AI systems not merely more intelligent, but more trustworthy.

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

When AI Knows It Doesn’t Know: Building Safer Robotic Systems with Confidence and Fallbacks

Ana Clara dos Santos

Introduction

1. The Problem with AI Predictions

2. From Prediction to Decision

3. A Simple Confidence Policy

4. Designing the System

Edge

Cloud

5. Where Amazon SageMaker AI Fits

6. Building the Decision Engine

7. Making the Fallback More Intelligent

8. Adding Sensor Quality to the Decision

9. Connecting the Robot to AWS IoT Core

10. Monitoring AI Confidence

11. Confidence Is Not the Same as Probability of Being Correct

12. Testing the Architecture

13. Edge vs. Cloud: Why Not Just Send Everything to the Cloud?

14. A Complete AWS-Oriented Architecture

15. Extending the Idea to ROS 2

16. What About Generative AI?

17. Failure Is a First-Class State

18. Limitations

Confidence calibration

Threshold selection

Sensor fusion

Adversarial and unexpected inputs

Hardware failures

Safety certification

Human factors

19. The Broader Idea

Conclusion