Let your Strands agent decide when to run Amazon Bedrock batch inference to save cost

Real-time inference is the costliest way to answer work that is not urgent. See how a Strands agent decides when to submit an Amazon Bedrock batch inference job instead, owns the async timing, and returns summarized results. Includes crisp boto3 code for submit, bounded polling, and results, plus prompt caching and inference profiles for further savings and cost visibility.

AK

Published Jun 4, 2026

Last modified Jun 4, 2026

Cut inference cost in your agents with Amazon Bedrock batch inference

This post shows how to give a Strands agent the
ability to decide when to run a workload as an Amazon Bedrock batch
inference job instead of calling the model in real time, and why letting the
agent make that call is more useful than a static script.

Why batch

Amazon Bedrock batch inference processes many prompts from a single JSONL file
in Amazon S3 and writes the results back to S3. In exchange for giving up
real-time responses, batch inference is offered at a lower price (50% lower) per token than
on-demand invocation (see Amazon Bedrock
pricing for current rates). For large,
latency-tolerant workloads, that difference is the main reason to use it.

Typical workloads that fit batch:

Summarizing or classifying a backlog of documents, tickets, or reviews.
Generating product descriptions or metadata for a catalog.
Offline evaluation and red-teaming over large prompt sets.
Nightly or scheduled enrichment jobs.

Why an agent, not a script

A shell script can submit a batch job. The script cannot decide whether a job
should be batched. That decision depends on context the agent already has:

Volume. A handful of prompts belongs in real-time inference. Thousands of
prompts belong in batch.
Urgency. If the user needs the answer now, the agent invokes the model
directly. If the result feeds a report due tomorrow, the agent submits a batch
job and moves on.
Orchestration. The agent can prepare the input, submit the job, poll on a
bounded schedule, fetch results, summarize them, and report back — within one
conversation, handling the timing on the user's behalf.

The agent shines on the complex path: turning an open-ended request
("classify these 5,000 reviews") into the right execution mode without the user
choosing batch versus real time. That is the first objective — use batch in
the agent when possible, and let the agent own the execution timing.

Considerations

Batch is not real time

Batch jobs are asynchronous. A job moves through Submitted → InProgress →
Completed, and completion can take minutes to hours depending on size and
service capacity. Design for this:

Do not block a conversation waiting for a job. Poll on a bounded budget and
return control.
Persist the job ARN so a later turn can resume.
Treat PartiallyCompleted as a valid outcome and inspect the error records.

Each model also enforces a minimum record count per job (often 100). Below the
minimum, the job is rejected at submission.

For production, prefer event-driven notifications over polling: see Monitor
Amazon Bedrock job state changes using Amazon EventBridge .
The bounded wait_for_job loop in this post keeps the sample self-contained, but
EventBridge removes the need to poll at all.

Feature constraints

Batch inference does not support tool calling (function calling), structured
output (response_format), or multi-turn interaction. Each record in the input
file is processed independently, so an agent that relies on those features at
inference time must use real-time invocation instead. Batch inference is also
not supported for provisioned models.

Model support constraints

Batch inference is supported on a subset of models and AWS Regions, and the
supported set differs from on-demand inference. Before the agent submits a job,
confirm the target model and Region support batch and that model access is
enabled in the account. For the current list, see Supported Regions and models for batch inference .
The input record schema is also model-specific (Amazon Nova uses messages-v1);
match modelInput to the model you submit against.

Quotas for batch inference

Batch inference enforces service quotas that can reject or queue a job. Review
them before sizing a workload, and request increases where the quota is
adjustable. Quotas to plan around include:

Batch inference input file size (in GB) for <Model> — the maximum size
of a single file submitted for batch inference for that model (for example,
Claude Sonnet 4.6). Split large inputs into multiple files or jobs to stay
under the limit.
Records per input file and minimum records per job — the per-file
record ceiling and the per-job floor (often 100).
Concurrent batch inference jobs per model — caps how many jobs run at
once; additional submissions queue.

View and manage current values in the Amazon Bedrock service quotas console.

How it works

The agent exposes the batch workflow as tools backed by boto3. The model picks
the tools and the order based on the request. A Strands tool is a plain Python
function with the @tool decorator; the docstring and type hints are what the
model reads to decide when and how to call it.

Tool	Bedrock / S3 call	Purpose
`configure_batch`	--	Write bucket, region, model to config
`prepare_input`	`s3:PutObject`	Build Nova JSONL, upload to S3
`submit_batch_job`	`CreateModelInvocationJob`	Submit the job, persist the ARN
`wait_for_job`	`GetModelInvocationJob`	Bounded poll for completion
`fetch_results`	`s3:GetObject`	Download outputs from S3
`summarize_batch_results`	--	Parse counts and sample outputs

Format the input

Each line is a record with a recordId and a modelInput in the model's native
schema. For Amazon Nova the input is the messages-v1 body. The prepare_input
tool builds one record per prompt and uploads the file to Amazon S3:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
from strands import tool

def _build_jsonl(prompts: list[str], dest: Path) -> int:
    """Write Nova-shaped batch records (recordId + modelInput) to dest."""
    width = max(4, len(str(len(prompts))))
    dest.parent.mkdir(parents=True, exist_ok=True)
    with dest.open("w", encoding="utf-8") as fh:
        for i, prompt in enumerate(prompts, start=1):
            rec = {
                "recordId": f"req-{i:0{width}d}",
                "modelInput": {
                    "schemaVersion": "messages-v1",
                    "system": [{"text": "You are a concise, helpful assistant."}],
                    "messages": [{"role": "user", "content": [{"text": prompt}]}],
                    "inferenceConfig": {
                        "maxTokens": 512, "temperature": 0.2, "topP": 0.9,
                    },
                },
            }
            fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
    return len(prompts)

@tool
def prepare_input(prompts: list[str] | None = None) -> dict:
    """Build the Nova batch JSONL and upload it to S3.

    Args:
        prompts: One prompt per record. Bedrock enforces a per-model minimum
                 record count (often 100); a low count is uploaded but flagged.
    """
    paths = _resolved_paths()                       # bucket, prefixes, keys
    if not paths["bucket"]:
        return {"ok": False, "error": "BUCKET not set. Call configure_batch first."}

    loaded = prompts or _load_prompts()             # fall back to prompts.txt
    local = INPUT_DIR / paths["input_object"]
    count = _build_jsonl(loaded, local)

    try:
        _client("s3").upload_file(str(local), paths["bucket"], paths["input_key"])
    except Exception as exc:
        return {"ok": False, "error": f"S3 upload failed: {exc}"}

    result = {"ok": True, "recordCount": count, "s3Uri": paths["input_s3"]}
    if count < 100:
        result["warning"] = "fewer than 100 records; most Nova models reject smaller jobs."
    return result

Submit the job

submit_batch_job reuses an existing batch service role and the S3 locations
from configuration, then calls CreateModelInvocationJob and persists the job
ARN so a later turn can resume:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
@tool
def submit_batch_job() -> dict:
    """Create a Bedrock batch inference job and persist its ARN."""
    paths = _resolved_paths()
    if not paths["bucket"] or not paths["model_id"]:
        return {"ok": False, "error": "BUCKET and MODEL_ID required."}

    role_arn = _resolve_role_arn()                  # config -> .role_arn -> IAM
    if not role_arn:
        return {"ok": False, "error": "no service role found. Run `make role` once."}

    job_name = f"{paths['job_name_prefix']}-{time.strftime('%Y%m%d-%H%M%S')}"
    try:
        resp = _client("bedrock").create_model_invocation_job(
            jobName=job_name,
            roleArn=role_arn,
            modelId=paths["model_id"],              # e.g. amazon.nova-lite-v1:0
            timeoutDurationInHours=paths["timeout_hours"],
            inputDataConfig={"s3InputDataConfig": {
                "s3Uri": paths["input_s3"], "s3InputFormat": "JSONL"}},
            outputDataConfig={"s3OutputDataConfig": {"s3Uri": paths["output_s3"]}},
        )
    except Exception as exc:
        return {"ok": False, "error": f"CreateModelInvocationJob failed: {exc}"}

    job_arn = resp.get("jobArn", "")
    JOB_ARN_FILE.write_text(job_arn + "\n", encoding="utf-8")
    return {"ok": True, "jobArn": job_arn, "jobName": job_name}

Poll without blocking

wait_for_job caps the wait so the agent never hangs. If the job is still
running at the deadline, it returns the last-seen status with terminal: false,
and the agent can wait again on a later turn:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
TERMINAL_STATES = {"Completed", "Failed", "Stopped", "Expired", "PartiallyCompleted"}

@tool
def get_job_status(job_arn: str = "") -> dict:
    """Return status of the batch job (defaults to the persisted ARN)."""
    arn = _resolve_job_arn(job_arn)
    if not arn:
        return {"error": "no job ARN known. Call submit_batch_job first."}
    resp = _client("bedrock").get_model_invocation_job(jobIdentifier=arn)
    return {"jobArn": arn, "status": resp.get("status"), "message": resp.get("message")}

@tool
def wait_for_job(job_arn: str = "", max_wait_seconds: int = 600,
                 interval_seconds: int = 30) -> dict:
    """Poll until terminal or the wait budget runs out (bounded)."""
    arn = _resolve_job_arn(job_arn)
    deadline = time.monotonic() + max(0, max_wait_seconds)
    while True:
        last = get_job_status(arn)
        if last.get("status") in TERMINAL_STATES:
            return {**last, "terminal": True}
        if time.monotonic() >= deadline:
            return {**last, "terminal": False,
                    "note": "still running; call wait_for_job again to keep waiting."}
        time.sleep(max(1, interval_seconds))

Read the results

Outputs land under s3://<bucket>/<output-prefix>/<job-id>/ as *.jsonl.out.
fetch_results downloads them; summarize_batch_results parses each line,
which echoes the recordId and adds modelOutput, or an error object for
failed records:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
@tool
def fetch_results(job_arn: str = "") -> dict:
    """Download a job's output files from S3 to output/<job-id>/."""
    arn = _resolve_job_arn(job_arn)
    paths = _resolved_paths()
    job_id = arn.rsplit("/", 1)[-1]
    prefix = f"{paths['output_prefix']}{job_id}/"
    local_dir = OUTPUT_DIR / job_id
    local_dir.mkdir(parents=True, exist_ok=True)

    s3 = _client("s3")
    downloaded = []
    for page in s3.get_paginator("list_objects_v2").paginate(
            Bucket=paths["bucket"], Prefix=prefix):
        for obj in page.get("Contents", []):
            key = obj["Key"]
            s3.download_file(paths["bucket"], key, str(local_dir / Path(key).name))
            downloaded.append(key)
    return {"ok": bool(downloaded), "outputDir": str(local_dir), "files": downloaded}

@tool
def summarize_batch_results(path: str = "") -> dict:
    """Count successes and errors, and sample a few outputs."""
    target = Path(path) if path else _find_latest_output_file()
    ok = errors = 0
    sample = []
    for line in target.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        rec = json.loads(line)
        if "error" in rec:
            errors += 1
            continue
        ok += 1
        if len(sample) < 3:
            text = rec["modelOutput"]["output"]["message"]["content"][0]["text"]
            sample.append({"recordId": rec["recordId"], "text": text[:400]})
    return {"ok": ok, "errors": errors, "sample": sample}

Register the tools with the agent

The model chooses tools from the list and calls them in whatever order the
request implies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from strands import Agent
from strands.models import BedrockModel

def build_agent() -> Agent:
    model = BedrockModel(
        model_id="us.amazon.nova-lite-v1:0",
        region_name="us-east-1",
        temperature=0.2,
        max_tokens=2048,
    )
    return Agent(
        model=model,
        tools=[
            configure_batch, prepare_input, submit_batch_job,
            get_job_status, wait_for_job, fetch_results,
            summarize_batch_results,
        ],
        system_prompt=(
            "You drive the Bedrock batch inference workflow for Amazon Nova. "
            "The IAM role is reused, not created. Standard order: configure_batch, "
            "prepare_input, submit_batch_job, wait_for_job (call again if not "
            "terminal), fetch_results, summarize_batch_results. Cite recordIds."
        ),
    )

Drive it from one prompt

1
2
python agent.py "Classify these reviews offline to save cost: prepare_input, \
  submit_batch_job, wait_for_job, fetch_results, then summarize_batch_results"

The agent decides batch is appropriate, submits the job, polls within budget,
and reports the summary — without the user managing any of the steps.

Summary

For latency-tolerant, high-volume work, batch inference lowers cost per token
versus real-time invocation. The value of putting it inside an agent is the
decision: the agent recognizes when a request is a batch workload, owns the
asynchronous timing, and returns a summarized result — work that is awkward to
encode in a static script.

# amazon-bedrock-agentcore # amazon-bedrock # strands-agents # cost-savings # cost-optimization

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Let your Strands agent decide when to run Amazon Bedrock batch inference to save cost

AK

Cut inference cost in your agents with Amazon Bedrock batch inference

Why batch

Why an agent, not a script

Considerations

How it works

Format the input

Submit the job

Poll without blocking

Read the results

Register the tools with the agent

Drive it from one prompt

Summary