Replaying and Validating Browser Automations with Amazon Nova Act

Discover how you can replay trajectories with Amazon Nova Act to bring testing rigor to AI-powered browser automation. Record browser sessions as JSON artifacts, replay them in fresh environments, and automatically validate URLs, screenshots, and DOM structures — without re-invoking the AI model. With configurable tolerances, strict/permissive modes, and an interactive HTML report, catch UI regressions, detect flakiness, and verify environment parity across deployments.

Shruti Arora

Published Apr 20, 2026

Last modified Apr 21, 2026

Introduction

Co-authored with Sakthivel Chellapparimanam

AI-powered browser automation is powerful but non-deterministic. The same natural language prompt can produce different action paths across runs. A workflow that navigates a website, clicks buttons, and fills forms might silently break when the UI changes; a network hiccup alters page load timing, or the AI agent takes a slightly different path. Traditional test assertions with hardcoded selectors don't apply here. You need a way to record what happened and verify it happens the same way again.

Amazon Nova Act 's SDK outputs trajectory artifacts from your browser automation sessions and one powerful thing you can do with them is build your own replay and validation pipeline. By saving a recorded session and replaying it later, you can automatically compare the two executions step by step, checking URLs, screenshots, and DOM structure to catch regressions or unexpected changes. Note that this replay and validation approach is provided as experimental sample code rather than a built-in SDK feature, so it's intended as a starting point you can adapt for your own use.

In this post, we walk through a complete example: recording a workflow, replaying it, and generating an HTML validation report. All code examples are from the nova-act-samples GitHub repository.

What is Nova Act?

Amazon Nova Act is an AI-powered UI automation tool that enables natural language control of applications. Instead of writing traditional automation scripts with brittle selectors and hardcoded coordinates, you describe what you want to accomplish in plain English, and Nova Act's AI vision model observes the interface, understands the intent, and executes the actions.

What is a Trajectory?

Trajectory is the data that Nova Act temporarily stores to maintain historical context while executing a workflow, including the input prompt, screenshots, and agent response. After an act() finishes, it will output traces of what it did in a self-contained html file. The location of the file is printed in the console trace.

1
** View your act run here: /var/folders/6k/75j3vkvs62z0lrz5bgcwq0gw0000gq/T/tmpk7_23qte_nova_act_logs/15d2a29f-a495-42fb-96c5-0fdd0295d337/act_844b076b-be57-4014-b4d8-6abed1ac7a5e_output.html

Learn more about storing these logs in your desired location here .

Trajectory replay

The sample code in the nova-act-samples GitHub repository provides a wrapper around the Nova Act SDK that compares the following 5 key attributes of a trajectory to identify whether the user interface (UI) has changed significantly or not.

Active URL – the browser URL at the time of the action
Screenshot – a base64-encoded image of the browser viewport
Simplified DOM – a lightweight representation of the page structure
Actions – the compiled sequence of actions Nova Act executed (clicks, keystrokes, scrolls)
Think statement – the model's reasoning output

Trajectories are serialized as JSON files and accessible in the session logs directory, making them portable and easy to inspect and since they use the same format the SDK expects at runtime, replay works by feeding those recorded actions back through the same SDK methods that would normally process live model output, effectively re-driving the browser as if the model were running again. To enable this functionality one needs to simply set the replayable attribute in the Nova Act instantiation function to true as shown below:

1
2
3
4
 with NovaAct(
    starting_page="https://nova.amazon.com/act/gym/next-dot",
    replayable=True,
)

A key distinction: replay does not invoke the model endpoint; it re-executes the recorded program of actions directly via the SDK's program runner. The same clicks, keystrokes, and scrolls are performed in the same order. The validators then check whether the application responded the same way.

Now, let’s understand the different validations that execute when you re-run the Nova Act workflow after setting replayable=True.

URL validation

Compares the current URL against the recorded URL, checking scheme, hostname, and path. Uses exact matching (0% tolerance) by default. If you're on the wrong page, that's a hard failure.

Screenshot validation

Compares the current browser screenshot against the recorded one using pixel-level image comparison. Allows a configurable tolerance (default: 10%) to account for minor rendering differences like timestamps, animations, or anti-aliasing. You can also restrict comparison to a specific bounding box region.

DOM validation

Compares the simplified DOM structure using sequence matching. With a default 10% tolerance, this catches significant structural changes (missing elements, reordered sections) while tolerating minor text content differences.

Running it yourself

Let’s try and run a sample to see this in action.

Recording a Trajectory

At first, you would run the Nova Act workflow such that Here's how to record a trajectory using save.py :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
from pathlib import Path 
from nova_act import NovaAct, workflow 
from examples.nova_act_client import NovaActClient 
from examples.utils import get_logger 
 
LOGGER = get_logger(__name__) 
 
@workflow(**NovaActClient.get_workflow_kwargs()) 
def main(logs_directory: str = "logs/") -> None: 
    Path(logs_directory).mkdir(parents=True, exist_ok=True) 
    with NovaAct( 
        starting_page="https://nova.amazon.com/act/gym/next-dot", 
        replayable=True, 
        logs_directory=logs_directory, 
    ) as nova: 
        result = nova.act("Go to the Destinations page") 
        LOGGER.info(f"Trajectory saved to: {result.trajectory_file_path}")

A couple of things to note from the above code:

replayable=True: This parameter enables trajectory capture and no additional configuration or instrumentation is needed.
result.trajectory_file_path: This gives you the path to the saved trajectory JSON directly from the ActResult.

To run the above code execute the following command:

1
python -m examples.trajectory.save

After the execution completes, the output prints the trajectory file path; pass this to the replay step.

1
2026-04-17 17:10:39,676 - __main__ - INFO - Trajectory saved to: /Users/sarora/Documents/Documents/AGI/NovaActSamples/examples/trajectory/logs/019d9d47-a852-77d6-a8b8-b51e847ac4c9/act_019d9d47-db0c-7159-9528-bf7052734743_Go_to_the_Destinations_page_trajectory.json

Replaying with Validation

The real value comes when you replay a recorded trajectory in a fresh browser session and validate that the browser reaches the same state.

Here's what a trajectory JSON file looks like (large fields truncated for readability):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
{ 
  "sdk_version": "3.2.216.0", 
  "prompt": "Go to the Destinations page", 
  "steps": [ 
    { 
      "active_url": "https://nova.amazon.com/act/gym/next-dot", 
      "image": "data:image/jpeg;base64,...<base64-encoded screenshot>...", 
      "simplified_dom": "<body scrollable scrolled-from-top='0%'>\n  <div>\n    <a nova-act-id='15'>...</a>\n    ...\n  </div>\n</body>", 
      "program": { 
        "calls": [ 
          { 
            "name": "think", 
            "kwargs": { 
              "value": "I see a Destinations button at the top navigation bar. I should click it." 
            } 
          }, 
          { 
            "name": "agentClick", 
            "kwargs": { "box": "<box>32,503,55,595</box>" } 
          }, 
          { "name": "waitForPageToSettle", "kwargs": {} }, 
          { "name": "takeObservation", "kwargs": {} } 
        ] 
      } 
    } 
  ], 
  "metadata": { 
    "session_id": "019d1ee5-e369-7e0f-a6dc-a1bdd3f2b2d3", 
    "act_id": "019d1ee6-0693-721b-88b2-e22d4d41a74a", 
    "num_steps_executed": 2, 
    "prompt": "Go to the Destinations page" 
  } 
}

Each step captures the full browser state (URL, screenshot, DOM) and the exact program of actions the model executed – clicks, keystrokes, waits – so replay can re-execute them without re-invoking the AI.

Here's replay.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
 
from nova_act import NovaAct, workflow 
from examples.nova_act_client import NovaActClient 
from examples.trajectory.trajectory_replay import load_trajectory, replay_trajectory 
from examples.utils import get_logger 
 
LOGGER = get_logger(__name__) 
 
@workflow(**NovaActClient.get_workflow_kwargs()) 
def main(trajectory_file: str, strict: bool = True) -> None: 
    trajectory = load_trajectory(trajectory_file) 
    LOGGER.info(f"Loaded trajectory: prompt='{trajectory.prompt}', steps={len(trajectory.steps)}") 
 
    page = trajectory.steps[0].active_url 
    with NovaAct(starting_page=page, replayable=True) as nova: 
        replay_trajectory(nova, trajectory, strict=strict) 
 
    LOGGER.info("Trajectory replay completed successfully")

Run it with the trajectory file from the save step:

1
python -m examples.trajectory.replay --trajectory_file=logs/act_<id>_trajectory.json

During replay, the TrajectoryRunner , in the runner.py file, re-executes each step's recorded program and runs three validators against the current browser state. Now, let’s dive deeper into the code and logs to get a better understanding.

Strict vs. Permissive Mode

The strict parameter on replay.py (and ReplayableNovaAct) controls how validation failures are handled:

strict=True (default): Any validation failure raises an InvalidTrajectoryReplay error, halting execution. Use this in CI pipelines or regression testing where failures should be blocked.
strict=False: Validation failures are logged as warnings, but execution continues. Use this for exploratory runs or when you want a complete report even if some steps diverge.

Automatic Save and Replay

For workflows with multiple act()calls, auto_replay.py provides ReplayableNovaAct that is a NovaAct subclass which overrides act() to automatically save each trajectory on first run and replay from the saved file on subsequent runs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
class ReplayableNovaAct(NovaAct): 
    """NovaAct subclass that automatically saves and replays trajectories.

    On first call to act() with a given prompt, runs the act normally and copies
    the trajectory to a stable trajectories/ directory. On subsequent calls,
    replays the saved trajectory instead.
    """

    def __init__(self, *args, strict: bool = True, **kwargs):
        kwargs["replayable"] = True
        super().__init__(*args, **kwargs)
        self._strict = strict
        self._act_index = 0
        TRAJECTORY_DIR.mkdir(parents=True, exist_ok=True)

    def act(self, prompt: str, **kwargs) -> ActResult | None:
        """Run an act or replay from a saved trajectory.

        On first run, executes the act and saves the trajectory file. On subsequent
        runs, detects the saved file and replays it with validation.

        Args:
            prompt: The natural language instruction for the act.
            **kwargs: Additional keyword arguments passed to NovaAct.act().

        Returns:
            ActResult on first run, None during replay.
        """
        pattern = str(TRAJECTORY_DIR / f"{self._act_index:03d}_*.json")
        matches = glob.glob(pattern)
        self._act_index += 1

        if matches:
            LOGGER.info(f"Replaying trajectory: {Path(matches[0]).name}")
            trajectory = load_trajectory(matches[0])
            replay_trajectory(self, trajectory, strict=self._strict)
            return None

        LOGGER.info(f"Running act: {prompt}")
        result = super().act(prompt, **kwargs)

        trajectory_path = result.trajectory_file_path
        if trajectory_path:
            filename = f"{self._act_index - 1:03d}_{Path(trajectory_path).name}"
            shutil.copy2(trajectory_path, TRAJECTORY_DIR / filename)
            LOGGER.info(f"Saved trajectory: {filename}")
        else:
            LOGGER.warning(f"No trajectory file for act: {prompt}")

        return result

Now, write your workflow logic as usual:

1
2
3
4
5
6
7
8
@workflow(**NovaActClient.get_workflow_kwargs())  
def main(strict: bool = True) -> None:  
    with ReplayableNovaAct(  
        starting_page="https://nova.amazon.com/act/gym/next-dot",  
        strict=strict,  
    ) as nova:  
        nova.act("Go to the Destinations page")  
        nova.act("View the details of the first destination")

The first run executes the act and saves the trajectory file. On subsequent runs, detects the saved file and replays it with validation.

Deep dive into the HTML Report

After each replay, an interactive HTML report is automatically generated in the session logs directory. The report includes:

Summary statistics: total steps, validations passed/failed, broken down by type (URL, image, DOM)
Side-by-side screenshots: expected (from recording) vs. observed (from replay) for each stepPer-step validation details: side-by-side screenshot comparison (expected vs. observed), difference percentages, and threshold values for each validator
Expandable diffs: for failed DOM or URL validations, you can expand to see the full expected and observed values
Color-coded status: green for passing validations, red for failures; making it easy to spot issues immediately

The report aboce shows a single step with 3/3 validations passed (100%). Each step displays expected vs. observed screenshots side-by-side, with URL, image, and DOM validation results including difference percentages and thresholds.

What are custom validators?

The validation system is extensible. Each validator inherits from ValidatorBase and implements two methods: validate()that returns a ValidationResult and handle_result() that decides whether to warn or raise. You can swap in your own validators for any of the three dimensions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from examples.trajectory.trajectory_replay.validators import ValidatorBase, ValidationResult 
 
class StrictUrlValidator(ValidatorBase): 
    """URL validator that also checks query parameters.""" 
 
    def validate(self, reference: str, observed: str) -> ValidationResult: 
        # Your custom comparison logic 
        ... 
 
    def handle_result(self, result: ValidationResult) -> None: 
        if not result.passed: 
            self.warn_or_raise(f"URL mismatch: {result.expected} != {result.observed}") 
 
# Use during replay 
replay_trajectory(nova, trajectory, url_validator=StrictUrlValidator())

What are some common use cases?

Trajectory Replay opens several practical workflows:

Regression testing: Record a golden trajectory for critical user flows, replay after deployments to catch UI regressions
Flakiness detection: Replay the same trajectory multiple times to identify non-deterministic behavior
Environment validation: Record in staging, replay in production (or vice versa) to verify environment parity
Debugging: When an automation fails, replay its trajectory step-by-step to pinpoint exactly where the divergence occurred

Key Considerations

SDK version sensitivity – Trajectories are tied to the SDK version that recorded them. A version mismatch triggers a warning (or error if strict_sdk_version=True on TrajectoryRunner).
Page load timing – The replay runner waits 5 seconds (wait_before_replay_ms=5000.0) before taking the first observation, giving pages with client-side rendering time to settle.
Dynamic content – Timestamps, ads, or personalized content will cause image/DOM validation noise. Use strict=False or custom validators with higher tolerances for pages with dynamic elements

Conclusion

Trajectory Replay applies to the rigor of traditional test assertions to AI-powered browser automation. By capturing past interactions and validating they play out the same way every time, you can build automation workflows with confidence and catch regressions before they reach your customers.

# agents # agentic-ai # ai # llm # all-builders-welcome

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Replaying and Validating Browser Automations with Amazon Nova Act

Shruti Arora

Introduction

What is Nova Act?

What is a Trajectory?

Trajectory replay

Running it yourself

Deep dive into the HTML Report

What are custom validators?

What are some common use cases?

Key Considerations

Conclusion