AWS Bedrock - Case Study on building SLMs

AWS Bedrock is a fully managed layer for building, customizing, and operating LLM apps—so product teams can ship value without babysitting GPUs, containers, or vector infra. It unifies access to many top models, adds opinionated building blocks (guardrails, evals, agents, knowledge bases), and offers both on-demand and provisioned throughput when you’re ready to scale.

1) Why Bedrock makes building LLM apps easier

Unified API across many FMs (Anthropic Claude, Meta Llama, Mistral, Amazon Nova/Titan, more) plus SDKs. You focus on prompts and flows, not per-vendor quirks.

Serverless inference choices: on-demand (token-based), batch for cheaper large jobs, latency-optimized (public preview) for snappier UX, and provisioned throughput for guaranteed capacity. There’s also Custom Model Import to host your own OSS architectures on-demand.

Built-ins for production:

Guardrails to standardize safety/privacy across models (including ones outside Bedrock).

Evaluations to compare models, and even assess RAG retrieval quality.

Agents / AgentCore (preview) to orchestrate multi-step tool calls and multi-agent workflows.

Knowledge Bases to stand up a managed RAG layer that handles ingestion, chunking, embeddings, and vector stores.

2) Our example: fine-tuning a small Llama-3 for Aspect-Based Sentiment Analysis (ABSA)

Goal
Given free-text reviews/tickets, classify 60 business aspects (e.g., pricing, onboarding, SLA, search quality, mobile UX, recruiter responsiveness, invoice accuracy) and attach sentiment for each present aspect.

Why a small Llama-3
We wanted a compact model (fast/cheap) that we could specialize for tight, domain-specific labels and long-tail phrasing. Bedrock supports fine-tuning Meta Llama 3.x/3.1/3.2 instruct variants, so we picked the smallest that met our accuracy/latency targets.

Data & format

~N labeled examples in JSONL with instruction-style SFT pairs:

input: raw text + schema reminder (list of 60 aspects + output JSON schema)

output: normalized JSON like {"aspects":[{"name":"pricing","sentiment":"negative"}, ...]}

Balance classes; add hard negatives (no aspect present).

Include few “adversarial” phrasings and domain slang to improve robustness.

Training on Bedrock

Upload training/validation JSONL to S3, start Model Customization (fine-tuning) on the chosen Llama-3 Instruct small variant in us-east-1 or us-west-2 (current FT regions). Tune epochs, LR, and early-stop on macro-F1.

Evaluate with Bedrock Model Evaluations (automatic + human) on a held-out set; track macro-F1, per-aspect F1, and calibration.

Serving & rollout

Deploy the custom model behind Bedrock’s managed endpoint (on-demand or provisioned throughput, depending on SLOs). IAM policies gate who can call it; CloudWatch for ops.

Standardize the output contract (JSON schema) so downstream analytics and dashboards stay stable across versions.

> Typical outcome: consistent uplift vs. zero-shot prompts on base models, especially for rare aspects and nuanced sentiment. (Exact gains vary by domain/dataset size.)

3) Company-wide hosting & “generate aspects + sentiment” at scale

We exposed a single analyze_aspects API internally:

1. Input: free text (ticket, review, email thread).

2. Optional RAG enrichment (if needed): Bedrock Knowledge Bases retrieves policy/FAQ snippets from Amazon OpenSearch (managed cluster or serverless) to ground tricky judgments. KB automates ingestion, chunking, embeddings, and vector writes to OpenSearch.

3. Inference: call our custom Llama-3 endpoint.

4. Output: JSON with [aspect, sentiment, confidence, evidence_spans].

Minimal Python for inference (Bedrock Runtime):

```python

import boto3, json
brt = boto3.client("bedrock-runtime", region_name="us-west-2")

model_id = "meta.llama3-8b-instruct-finetuned:company-aspects-v1" # example id
prompt = """
<|begin_of_text|><|start_header_id|>user<|end_header_id|>
Return JSON with aspects and sentiment for the text:
TEXT: "Pricing looked okay but the onboarding was chaotic and the mobile app kept crashing."
SCHEMA: [{"name": "...","sentiment": "positive|negative|neutral"}]
<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

resp = brt.invoke_model(
modelId=model_id,
body=json.dumps({"prompt": prompt, "max_gen_len": 512, "temperature": 0.1})
)
print(json.loads(resp["body"].read())["generation"])

```

Because it’s Bedrock, no model servers to run, and we can choose on-demand for bursty usage or provisioned throughput for steady, guaranteed TPS (or try batch for backfills).

4) Native OpenSearch connectivity (for RAG + analytics)

OpenSearch Serverless or Managed Clusters work as vector stores for Knowledge Bases; KB handles chunking/embeddings and keeps the text/metadata fields you need for grounding and traceability.

As of July 2025, KB also supports OpenSearch Managed Clusters directly (not just serverless), giving more tuning control when you need it.

This let us index tickets/reviews, ground the model on policy phrases, and log aspect/sentiment outputs back into OpenSearch for searchable analytics—end-to-end inside AWS.

5) Pain points & areas to improve

Regional constraints for customization: Fine-tuning currently runs in us-east-1 / us-west-2; plan for data residency and cross-region latency if your teams sit elsewhere.

Knowledge Bases + OpenSearch networking: Today, KB requires public access for OpenSearch Managed Clusters (no VPC-only domains yet), which may complicate security posture.

Pricing complexity: Choosing among on-demand, batch, latency-optimized, provisioned throughput, and custom-model import demands careful capacity modeling and guardrails. (Great flexibility—just budget the analysis time.)

Training knobs & transparency: Bedrock’s fine-tuning UX intentionally abstracts infra; advanced users sometimes want deeper control/telemetry than the managed path exposes (e.g., bespoke schedulers or custom tokenization flows).

Agentic stacks still evolving: AgentCore is powerful but in preview; expect APIs and best practices to keep shifting.

Takeaway

For teams that want fast iteration, standardized safety, and enterprise guardrails, Bedrock is a pragmatic default. Our ABSA case—fine-tuned small Llama-3, hosted centrally, integrated with OpenSearch—went from prototype to company-wide API without standing up any GPU clusters or vector infra. That’s the Bedrock advantage: build more, operate less.

#pyconindia2025