I Built a Simulation-First Safety Gate for Strands Robots: Reject Bad Episodes Before They Reach Real Hardware

The current Strands Robots project supports simulation by default, real-hardware execution, policy execution, recording, multiple simulation backends, and robot fleet/mesh capabilities. AWS also now has AgentCore Evaluations for online, on-demand, and batch evaluation of agent behavior.

Muhammad Umair Ashraf

Cloud Data Engineer

Published Oct 8, 2026

Last modified Oct 8, 2026

Introduction

One of the most interesting things about physical AI is that an AI agent can move from answering a question to controlling something in the real world.

That also changes the definition of a failure.

For a chatbot, a bad answer might waste a few seconds.

For a robot, a bad action can mean a collision, a dropped object, excessive joint movement, or damage to hardware.

That led me to a different question:

What if we don't ask the agent to decide whether an action is safe after it reaches the robot? What if we make the entire episode pass a safety gate before allowing it to run on physical hardware?

This article explores a simulation-first safety pattern using Strands Robots, recorded episodes, policy execution, and evaluation.

The goal is simple:

Simulation → Record → Evaluate → Reject/Approve → Real Hardware

1. The Problem: “It Worked” Is Not Enough

A robot policy can successfully complete a task once and still be unsafe.

For example:

1
2
Task:
Pick up the red cube and place it in the box.

The policy may succeed, but during the episode it could:

move a joint faster than expected
approach an object too aggressively
leave the permitted workspace
collide with another object
run longer than the allowed execution window
repeatedly oscillate before completing the task
finish the task while violating a safety constraint

So I wanted to separate two questions:

Task evaluation

Did the robot accomplish the task?

Safety evaluation

Did the robot accomplish the task without violating our constraints?

Those should not be the same metric.

2. Why Strands Robots Is a Good Fit

Strands Robots gives a Strands Agent a robot tool and supports simulation as the default workflow, while real hardware can be explicitly selected. The project also supports policy execution, recording, simulation, and multiple robot backends. (GitHub )

That creates an interesting development pattern:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
              DEVELOPMENT
                  │
                  ▼
          ┌───────────────┐
          │ Strands Agent │
          └───────┬───────┘
                  │
                  ▼
          ┌───────────────┐
          │   Simulation  │
          │    MuJoCo     │
          └───────┬───────┘
                  │
                  ▼
             Record Episode
                  │
                  ▼
           Safety Evaluation
             /           \
          FAIL             PASS
           │                │
           ▼                ▼
        Reject         Hardware Test

The important idea is that simulation becomes a quality gate, rather than just a demonstration environment.

3. The Architecture

My proposed architecture looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
User
  │
  ▼
Strands Agent
  │
  ▼
Robot Tool
  │
  ▼
Simulation Environment
  │
  ├── Observation
  ├── Actions
  ├── Robot State
  └── Episode Recording
          │
          ▼
   Safety Gate Evaluator
          │
    ┌─────┴─────┐
    │           │
  REJECT       PASS
    │           │
    ▼           ▼
  Review     Hardware
             Execution

This architecture is deliberately conservative.

The agent does not automatically move from simulation to hardware.

Instead:

A simulation episode earns the right to be tested on hardware.

4. Recording the Episode

Strands Robots supports recording episodes using its dataset recording capabilities. The documentation shows recording a policy run into a LeRobot v3 dataset. (GitHub )

A simplified example looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from strands_robots import Robot

robot = Robot("so100")

robot.start_recording(
    repo_id="my-user/safety-test",
    task="pick up the red cube",
    fps=50
)

robot.run_policy(
    robot_name="so100",
    instruction="pick up the red cube",
    policy_provider="mock",
    duration=10.0
)

robot.stop_recording()

The important part isn't the specific task.

It is the fact that we now have an episode that can be inspected and evaluated.

5. What Should the Safety Gate Check?

I would divide the evaluator into five independent checks.

Check 1 — Workspace Safety

Did the robot remain inside the permitted workspace?

1
2
3
x_min ≤ x ≤ x_max
y_min ≤ y ≤ y_max
z_min ≤ z ≤ z_max

If the end effector leaves the allowed region:

1
SAFETY = FAIL

Check 2 — Joint Velocity

A policy might technically complete a task while producing excessive movement.

For each joint:

1
velocity[joint] <= maximum_allowed_velocity

If:

1
max_velocity > threshold

the episode is rejected.

Check 3 — Collision / Contact

Collision behavior is especially important in physical robotics.

The evaluator should inspect whether the episode contains:

1
2
3
4
unexpected_contact
collision
self_collision
unsafe_contact

A task can therefore have:

1
2
Task Success = TRUE
Safety = FALSE

That distinction is critical.

6. Check 4 — Execution Time

A policy that normally completes a task in 5 seconds but suddenly requires 60 seconds deserves investigation.

For example:

1
2
3
4
MAX_EPISODE_SECONDS = 20

if episode_duration > MAX_EPISODE_SECONDS:
    return "REJECT"

This also helps identify policies that become trapped in repetitive behavior.

7. Check 5 — Action Stability

Another useful signal is action stability.

Suppose the robot produces:

1
2
3
4
5
6
7
left
right
left
right
left
right
...

for many control cycles.

The robot may technically be moving, but the policy isn't converging.

We can calculate an oscillation score:

1
2
oscillation_score =
    direction_changes / total_actions

If that value becomes unusually high, the episode can be flagged for review.

8. The Safety Decision

Instead of returning only:

1
SUCCESS

the evaluator should return structured information:

1
2
3
4
5
6
7
8
9
{
  "task_success": true,
  "workspace_safe": true,
  "velocity_safe": true,
  "collision_free": true,
  "execution_time_safe": true,
  "action_stability": true,
  "hardware_approval": true
}

This is much more useful than a single score.

9. A Simple Safety Evaluator

A simplified Python implementation could look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
def evaluate_episode(episode):
    violations = []

    if not workspace_is_safe(episode):
        violations.append("WORKSPACE_VIOLATION")

    if not joint_velocity_is_safe(episode):
        violations.append("JOINT_VELOCITY_LIMIT")

    if has_collision(episode):
        violations.append("COLLISION")

    if episode.duration > 20:
        violations.append("TIMEOUT")

    if action_oscillation_is_high(episode):
        violations.append("ACTION_OSCILLATION")

    return {
        "approved": len(violations) == 0,
        "violations": violations
    }

The key design principle is:

Fail closed.

If the evaluator cannot determine whether an episode is safe, it should not automatically approve the episode for hardware.

10. Adding an “Unknown” State

This is where I would take the idea further.

A normal evaluator has:

1
2
PASS
FAIL

But physical AI benefits from:

1
2
3
PASS
FAIL
UNKNOWN

For example:

1
2
3
4
{
  "decision": "UNKNOWN",
  "reason": "Insufficient evidence to verify collision-free execution"
}

The UNKNOWN state is important because absence of evidence is not evidence of safety.

The workflow becomes:

1
2
3
4
5
6
7
8
9
10
11
12
             Episode
                │
                ▼
          Safety Evaluator
                │
       ┌────────┼────────┐
       ▼        ▼        ▼
      PASS     FAIL    UNKNOWN
       │        │        │
       ▼        ▼        ▼
   Hardware   Reject   Human
    Test               Review

This is the safety-oriented version of the same philosophy behind “knowing when it does not know.”

11. Where Amazon Bedrock Fits

I would not use an LLM as the only safety mechanism.

Instead, I would divide the evaluation into two layers.

Deterministic layer

Use code for things that can be measured exactly:

1
2
3
4
5
6
Joint velocity
Workspace boundaries
Episode duration
Collision flags
Distance thresholds
Action limits

AI evaluation layer

Use a model-based evaluator for higher-level questions:

1
2
3
4
5
6
7
Did the robot appear to accomplish the requested task?

Was the behavior consistent with the task?

Was there unexpected behavior?

Does the episode require human review?

This is similar to modern agent evaluation patterns where deterministic checks and model-based evaluators are used for different failure modes. AWS's current AgentCore Evaluations supports built-in and custom evaluators and can evaluate agent traces using online, on-demand, and batch modes. (AWS Documentation )

12. AgentCore Evaluations for the Agent Layer

There is an important distinction here.

The robot policy controls physical movement.

The Strands Agent controls the reasoning and tool interaction around the robot.

Those are different things to evaluate.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
                    AGENT EVALUATION
                           │
                           ▼
                    Did the agent
                  select the right tool?
                           │
                           ▼
                    Did it issue the
                   correct instruction?
                           │
                           ▼
                    Did it achieve
                    the user goal?

AgentCore Evaluations supports metrics such as goal completion, correctness, tool selection, and custom evaluation dimensions. (AWS Documentation )

That means we can evaluate both layers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
┌──────────────────────────────┐
│       STRANDS AGENT          │
│                              │
│ Tool Selection               │
│ Instruction Quality          │
│ Goal Completion              │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│       ROBOT POLICY           │
│                              │
│ Workspace Safety             │
│ Velocity Safety               │
│ Collision                     │
│ Stability                     │
│ Execution Time                │
└──────────────┬───────────────┘
               │
               ▼
        HARDWARE APPROVAL

13. Why Batch Evaluation Is Useful

One episode isn't enough.

A policy might work on:

1
red cube

but fail on:

1
2
3
4
5
blue cube
different position
different lighting
different object orientation
different starting position

Strands Robots already supports recording datasets, while AgentCore Evaluations supports batch evaluation over multiple sessions. (GitHub )

So instead of asking:

“Did my robot work?”

I would ask:

“How consistently does my robot remain safe across a distribution of situations?”

14. Build a Safety Dataset

I would create test categories such as:

Category	Example
Normal	Pick cube from expected position
Position variation	Cube moved 10 cm
Orientation variation	Cube rotated
Workspace edge	Object near boundary
Occlusion	Partial camera obstruction
Long task	Multi-step manipulation
Failure	Object unavailable
Recovery	Object moved during task

Then evaluate every policy against the same dataset.

This makes policy comparisons reproducible.

15. The Release Gate

The final architecture becomes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
                 New Policy
                     │
                     ▼
              Simulation Tests
                     │
                     ▼
             Episode Recording
                     │
                     ▼
              Safety Evaluation
                     │
          ┌──────────┴──────────┐
          │                     │
        FAIL                   PASS
          │                     │
          ▼                     ▼
       Reject              Agent Evaluation
                                │
                                ▼
                         Quality Threshold
                                │
                     ┌──────────┴──────────┐
                     │                     │
                   FAIL                   PASS
                     │                     │
                     ▼                     ▼
                  Reject             Hardware Test

This creates a two-stage quality gate:

Gate 1

Is the robot behavior physically safe?

Gate 2

Is the agent behavior correct?

Only after both pass should the system move toward hardware execution.

16. What Happens When Something Fails?

I don't want the system to simply say:

1
Evaluation failed.

Instead, every failure should produce a structured reason.

For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
{
  "episode": "episode_042",
  "status": "REJECTED",
  "violations": [
    {
      "type": "JOINT_VELOCITY",
      "joint": "joint_3",
      "observed": 2.84,
      "limit": 2.0
    }
  ],
  "recommended_action": "Review policy checkpoint"
}

Now the failure becomes training data.

That creates a much more interesting loop:

1
2
3
4
5
6
7
8
9
10
11
Run
 ↓
Evaluate
 ↓
Find Failure
 ↓
Record Failure
 ↓
Improve Policy
 ↓
Run Again

17. From Safety Gate to Continuous Improvement

Once enough episodes are collected, the dataset becomes more than an evaluation set.

It becomes a learning asset.

1
2
3
4
5
6
7
8
9
10
11
12
13
Successful Episodes
        +
Failure Episodes
        +
Human Reviews
        ↓
Evaluation Dataset
        ↓
Policy Improvement
        ↓
New Policy
        ↓
Simulation

This creates a closed engineering loop rather than treating evaluation as the final step.

18. What I Like About the Simulation-First Approach

The biggest advantage is not simply avoiding hardware damage.

It changes the economics of experimentation.

You can test:

1
2
3
100 episodes
1,000 episodes
10,000 episodes

without physically repeating every experiment on a robot.

Strands Robots currently provides simulation backends including MuJoCo, and its ecosystem also includes Isaac Sim integration. (GitHub )

That makes simulation useful as a pre-hardware validation layer.

19. Important Limitation

Simulation is not reality.

A policy that passes 10,000 simulated episodes does not automatically become safe on a physical robot.

The real world introduces:

sensor noise
calibration errors
latency
mechanical variation
unexpected obstacles
communication failures
hardware wear
lighting differences
model mismatch

Therefore:

Simulation should reduce risk, not claim to eliminate it.

The first physical execution should still be controlled and supervised.

20. My Proposed Production Pattern

For a more mature system, I would use:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
                   USER
                     │
                     ▼
               STRANDS AGENT
                     │
              Amazon Bedrock
                     │
                     ▼
                ROBOT TOOL
                     │
             ┌───────┴───────┐
             │               │
          SIMULATION       HARDWARE
             │
             ▼
       EPISODE RECORDER
             │
             ▼
       SAFETY EVALUATOR
             │
      ┌──────┼──────┐
      │      │      │
     PASS   FAIL  UNKNOWN
      │      │      │
      │      │      └── Human Review
      │      │
      │      └───────── Policy Improvement
      │
      ▼
 Hardware Approval
      │
      ▼
 Controlled Execution
      │
      ▼
 AgentCore / CloudWatch
 Evaluation & Monitoring

AgentCore Evaluations can continuously monitor deployed agent behavior using sampled live traffic, while results can be written to CloudWatch for monitoring and analysis. (AWS Documentation )

21. The Bigger Idea

The interesting shift is from:

“Can an AI agent control a robot?”

to:

“Can we build a measurable system that knows when a robot behavior is safe enough to progress to the next stage?”

That is a much more important engineering problem.

A physical AI system should not have only:

1
Agent → Robot

It should have:

1
2
3
4
5
6
7
8
9
10
11
12
13
Agent
  ↓
Simulation
  ↓
Evaluation
  ↓
Safety Gate
  ↓
Human/Policy Review
  ↓
Hardware
  ↓
Continuous Monitoring

The robot gets autonomy gradually.

Final Takeaway

My main takeaway from this experiment is simple:

Don't make the physical robot your first test environment.

Use simulation to generate episodes.

Use deterministic checks for measurable safety constraints.

Use AI-based evaluation for higher-level behavior.

Introduce an explicit UNKNOWN state when the system lacks enough evidence.

Then promote only passing behavior toward hardware.

Strands Robots makes the simulation-to-hardware workflow practical by providing a common robot interface across simulation and physical execution, while AWS's current agent evaluation capabilities provide a path for measuring agent behavior before and after deployment. (GitHub )

The result is not an agent that simply acts.

It is an agent whose actions have to earn the right to reach the real world.

# aws-academy # aws-builders # aws-community # aws-community-builders # robotics

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

I Built a Simulation-First Safety Gate for Strands Robots: Reject Bad Episodes Before They Reach Real Hardware

Muhammad Umair Ashraf

Introduction

1. The Problem: “It Worked” Is Not Enough

Task evaluation

Safety evaluation

2. Why Strands Robots Is a Good Fit

3. The Architecture

4. Recording the Episode

5. What Should the Safety Gate Check?

Check 1 — Workspace Safety

Check 2 — Joint Velocity

Check 3 — Collision / Contact

6. Check 4 — Execution Time

7. Check 5 — Action Stability

8. The Safety Decision

9. A Simple Safety Evaluator

10. Adding an “Unknown” State

11. Where Amazon Bedrock Fits

Deterministic layer

AI evaluation layer

12. AgentCore Evaluations for the Agent Layer

13. Why Batch Evaluation Is Useful

14. Build a Safety Dataset

15. The Release Gate

Gate 1

Gate 2

16. What Happens When Something Fails?

17. From Safety Gate to Continuous Improvement

18. What I Like About the Simulation-First Approach

19. Important Limitation

20. My Proposed Production Pattern

21. The Bigger Idea

Final Takeaway