OS-Level Browser Control with Amazon Bedrock AgentCore: Automating Native Dialogs, and the Failure Modes That Bite
Amazon Bedrock AgentCore Browser's `InvokeBrowser` API drives mouse, keyboard, and full-desktop screenshots at the operating-system level, reaching the native dialogs that Chrome DevTools Protocol cannot. This post wires a real claims-portal workflow that attaches a file through a native chooser and dismisses a print dialog, then catalogs the failure modes — silent no-op keys, coordinate drift, session timeouts, and base64 corruption — with fixes for each.
OS-Level Browser Control with Amazon Bedrock AgentCore
Chrome DevTools Protocol has a hard boundary. It drives the DOM, the web layer that Playwright and
browser-use expose. Native print dialogs, OS file choosers, certificate prompts, right-click context menus, and Chrome settings chrome render outside that boundary, so CDP cannot see them and a coordinate-based agent has nothing to click.Amazon Bedrock AgentCore Browser's
InvokeBrowser API closes that gap. It dispatches both standard web automation over a WebSocket CDP endpoint and OS-level actions over a SigV4-signed REST endpoint (POST /browsers/{browserIdentifier}/sessions/invoke). The OS actions run on the full desktop of the session's isolated microVM, so the agent can observe native UI with a screenshot, reason about it, and act on it in the same session.This post builds one real workflow on that API, then focuses on where it breaks.
When to reach for OS actions instead of CDP
| Situation | CDP / Playwright | InvokeBrowser (OS) |
|---|---|---|
| Navigate, fill DOM forms, query the accessibility tree | Yes | Overkill |
window.print() opens a system print dialog | Invisible | Click / escape |
| "Attach receipt" opens an OS file chooser | Invisible | keyType path + enter |
Keyboard shortcut that triggers OS behavior (ctrl+p, ctrl+s) | Partial | keyShortcut |
| Right-click context menu rendered by the OS | No | mouseClick with RIGHT |
| Full-desktop screenshot including OS notifications | Viewport only | screenshot |
The rule of thumb: if the target is inside the DOM, stay on CDP. If the target is something the operating system painted, it is an
InvokeBrowser action.The eight actions
Every request carries exactly one member of the
BrowserAction union and returns a matching BrowserActionResult with a SUCCESS or FAILED status. The active session is identified by the x-amzn-browser-session-id header.| Action | Required | Optional | Notes |
|---|---|---|---|
mouseClick | — | x, y, button, clickCount | Defaults to current position, LEFT, single click. clickCount 1–10. |
mouseMove | x, y | — | Move cursor to coordinates. |
mouseDrag | startX, startY, endX, endY | button | Drag from start to end. |
mouseScroll | x, y | deltaX, deltaY | Range −1000..1000; negative deltaY scrolls down. |
keyType | text | — | Types a string. Max 10,000 chars. ASCII only. |
keyPress | key | presses | Press a key N times, 1–100. Key names lowercase. |
keyShortcut | keys | — | Chord of up to 5 keys, e.g. ["ctrl","a"]. |
screenshot | — | format | Full OS desktop, base64 PNG. Only action that returns data. |
Coordinate actions are bounded by the session viewport:
1 < x < viewportWidth-2 and 1 < y < viewportHeight-2. The default viewport is 1456×819 and is set at session start with viewPort.A UI-automation use case: file a claim on a legacy portal
A claims team uses a vendor portal that ships no API. The monthly job is: open claim #4812, fill three fields, attach a signed receipt PDF, and confirm the submission by dismissing a print dialog. The form is DOM, but the attach step opens a native file chooser and the submit step calls
window.print() — both outside CDP.Session setup with boto3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import boto3, time
REGION = "us-west-2"
BROWSER_ID = "aws.browser.v1"
cp = boto3.client("bedrock-agentcore-control", region_name=REGION)
dp = boto3.client("bedrock-agentcore", region_name=REGION)
# ~30 min budget so a human-in-the-loop step doesn't kill the run
session = dp.start_browser_session(
browserIdentifier=BROWSER_ID,
name="claims-4812",
sessionTimeoutSeconds=1800,
viewPort={"width": 1600, "height": 900},
)
sid = session["sessionId"]bedrock-agentcore-control manages browser resources (CreateBrowser, DeleteBrowser); bedrock-agentcore runs sessions (StartBrowserSession, InvokeBrowser, StopBrowserSession). Keep those two IAM surfaces separate — provisioning actions should not live on the agent execution role.The helper
1
2
3
4
5
6
7
8
9
10
11
12
13
def invoke(action: dict) -> dict:
return dp.invoke_browser(
browserIdentifier=BROWSER_ID,
sessionId=sid,
action=action,
)["result"]
def shot(path="screen.png"):
result = invoke({"screenshot": {"format": "PNG"}})
# boto3 auto-decodes the base64 payload: result["data"] is already bytes.
with open(path, "wb") as f:
f.write(result["screenshot"]["data"])
return pathThat comment is not cosmetic. Over the SigV4 REST endpoint the payload arrives base64-encoded; through boto3 the SDK decodes it for you. Calling
base64.b64decode() on the boto3 result writes a corrupt PNG. This exact gotcha showed up in the Dynatrace observability walkthrough (see Community Wisdom below).Steps 1–3: the native file chooser
Clicking "Attach receipt" opens an OS file dialog, which CDP cannot reach. Capture the screen, then drive the chooser with keyboard input:
1
2
3
4
5
# Click the DOM control via CDP first; the native chooser opens on top.
# Then, at the OS level, type the path and confirm.
invoke({"keyType": {"text": "/home/agent/receipts/claim-4812.pdf"}})
invoke({"keyPress": {"key": "enter"}})
shot("after-attach.png") # verify the filename appeared, then continueSteps 4–5: dismiss the print dialog
The submit handler calls
window.print() and a native print dialog appears. A vision model reads shot(), finds Cancel, and returns coordinates; the agent clicks them.1
2
3
4
invoke({"screenshot": {"format": "PNG"}})
# vision model returns e.g. {"x": 410, "y": 535}
invoke({"mouseClick": {"x": 410, "y": 535, "button": "LEFT"}})
shot("after-submit.png")Step 6: release the session
1
dp.stop_browser_session(browserIdentifier=BROWSER_ID, sessionId=sid)That is the whole pattern: action → screenshot → reaction. Every action returns only a status; the screenshot is how the agent learns what changed.
Failure modes, and how each one actually fails
The status field means "the event was dispatched," not "the UI did what you wanted." Most production incidents come from that distinction.
| # | Symptom | Root cause | Fix |
|---|---|---|---|
| 1 | ValidationException 400 on a click you copied from a screenshot | Coordinates outside the session viewport, even though the full-desktop PNG is larger | Set viewPort explicitly; derive click coordinates from that same width/height |
| 2 | Non-ASCII text silently missing | keyType is ASCII-only; Unicode is skipped, no error | Paste via CDP clipboard then keyShortcut ["ctrl","v"] |
| 3 | keyShortcut returns SUCCESS but nothing happened | No key-name validation; wrong case or unknown key is a silent no-op | Allowlist lowercase names (enter, esc→escape, ctrl); confirm with a screenshot |
| 4 | Click "succeeds" but hits the wrong control | SUCCESS only means dispatch; a toast, cookie banner, or lazy layout shifted the target | Re-screenshot after every action; never chain clicks from a stale frame |
| 5 | ResourceNotFoundException 404 mid-run | Session timed out (default 900 s) and the microVM was reclaimed | Raise sessionTimeoutSeconds (up to 8 h); heartbeat; reconnect on 404 |
| 6 | ThrottlingException 429 / InternalServerException 500 | API rate limit or transient backend failure treated as deterministic | Retry the whole action with exponential backoff and jitter, then re-screenshot |
| 7 | Corrupt PNG | Double base64-decoding the boto3 result | Write result["screenshot"]["data"] directly under boto3 |
| 8 | Vision model keeps clicking yesterday's coordinates | Screenshot captured before the UI repainted | Insert a settle delay or poll until two consecutive screenshots match |
| 9 | Some context-menu items do nothing | The browser runs in a virtualized sandbox; not every OS menu action is wired | Treat right-click as best-effort; fall back to a DOM route |
| 10 | Agent gets blocked by CAPTCHA | Anti-bot layers inspect TLS/HTTP/2 fingerprints and behavior before any DOM runs | Don't use the browser for general search; enable Live View for human handoff |
| 11 | Cost keeps growing after the task ends | Sessions bill while active; a crashed agent leaks them | Always StopBrowserSession in a finally; alert on long-lived sessions |
Silent failures deserve the most attention
Items 2 and 3 are the dangerous ones because the API returns
SUCCESS. A retry loop cannot catch them; only observation can. Build the loop so the model must confirm the effect, not the status: read the field value back, or diff two screenshots, before advancing. In benchmark work on LLM-in-control browser agents (Lucas Dantas, ANCHOR), the model declared success without completing the task in roughly one run in four — worse than a loud failure, because no one notices.Coordinates are a viewport contract, not a pixel guess
The screenshot is a full desktop capture; the clickable area is the session viewport. If you start a session at the default 1456×819 and then reason over a 1920×1080 screenshot, every coordinate below the fold or to the right is off by a ratio. Pin the viewport at session start and keep one coordinate space end to end.
Timeouts are the quiet killer for human-in-the-loop
AgentCore sessions default to 900 seconds and self-terminate. Any workflow that pauses for an approval opens a race with the clock; the next action returns
ResourceNotFoundException. Size sessionTimeoutSeconds to the worst-case human delay, and persist a browser profile so re-authentication does not restart the whole task.Observability makes the failures visible
An action-screenshot-reaction loop is a distributed system — LLM call, browser action, page load, screenshot, decision. Instrument it:
- Live View streams the session to a browser so a human can watch and take over during a CAPTCHA or an unexpected dialog.
- Session recording to S3 captures DOM replay, console logs, and network events for post-mortem.
- OpenTelemetry spans per action, exported over OTLP, correlate a failed click with the exact screenshot on screen when it happened. The Dynatrace walkthrough shows the span-per-action shape (
os-screenshot,os-mouse-click,os-key-shortcut) with status and coordinates as attributes.
One span per action, with the screenshot URI as an attribute, turns "the agent took 45 seconds" into "the native file chooser never received
enter."Security notes
Each session runs in a dedicated microVM with isolated CPU, memory, and filesystem; on completion the microVM is terminated and its memory sanitized. Every call is SigV4-signed, so unauthenticated requests are rejected with 403. Keep the execution role least-privilege —
InvokeBrowser, StartBrowserSession, StopBrowserSession — and never type credentials through keyType; prefer a persisted profile and hand off login to a human when a task needs one.What to take away
OS-level actions extend browser automation past the DOM boundary, and that is exactly why they fail differently. CDP failures usually raise;
InvokeBrowser failures often return SUCCESS and quietly do nothing. Design the loop to verify effects, pin the viewport, treat key names as an allowlist, and instrument every action. Do that, and native dialogs stop being the reason a workflow cannot ship.Community Wisdom
- "Observing AI Browser Agents with AgentCore Browser and Dynatrace" — Jason Mimick, AWS Builder Center. Documents the boto3 base64 auto-decode gotcha and a one-span-per-OS-action trace shape. https://builder.aws.com/content/3CM86Qd5NtcFa4iikmJpiXWhk3i/observing-ai-browser-agents-with-agentcore-browser-and-dynatrace
- "Why the LLM in my automation agent never clicks anything" — Lucas Dantas (dev.to). Measured false-success rates: LLM-in-control agents claimed success without completing the task in ~33–41% of runs; a plan/verify split cut it to ~2%. https://dev.to/lucasdantas2701/why-the-llm-in-my-automation-agent-never-clicks-anything-2edm
- "Why Browser Agents Fail in Production Without Semantic Layers" — Parvej Shah (dev.to). Ephemeral CSS-in-JS class names and opaque div trees break coordinate and selector targeting; stable
data-agentattributes make machine intent deterministic. https://dev.to/parvejshah/why-browser-agents-fail-in-production-without-semantic-layers-35hk - "Why Your AI Agent Gets Blocked and Your Chrome Doesn't" — Santiago Blaine (dev.to). Bot detection is a layered stack (TLS/JA3, HTTP/2 frames, headers) that fires before DOM scripts; a TLS fingerprint that contradicts the UA is an instant block. https://dev.to/santiago_blaine_f319c228c/why-your-ai-agent-gets-blocked-and-your-chrome-doesnt-57ch
- "Browser Tab Throttling for Playwright Agents" — mech_app_ai (dev.to). High concurrency triggers CAPTCHA walls and self-inflicted rate limits; queueing beats parallelism. https://dev.to/mech_app_ai/browser-tab-throttling-for-playwright-agents-why-50-concurrent-sessions-trigger-captchas-and-how-3cd8
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article