From User Query to Grounded Response: An Enterprise LLMOps Pipeline on AWS
Enterprise-grade multi-agent AI platform built with LangGraph, Amazon Bedrock, Titan Embeddings, OpenSearch Serverless, FastAPI, React, Docker, Amazon ECS Fargate, ALB, CloudFront, and production LLMOps practices including semantic RAG, AI guardrails, and real-time WebSocket streaming.
Introduction
Generative AI has rapidly evolved from simple conversational interfaces into intelligent systems capable of solving complex domain-specific problems. While traditional chatbot architectures are suitable for general-purpose interactions, enterprise healthcare applications demand significantly more than a single large language model responding to every request.
Healthcare systems process highly contextual information such as medical history, medications, appointments, clinical documentation, laboratory reports, and physician notes. Each of these domains requires specialized reasoning, structured retrieval, safety validation, and reliable orchestration.
Rather than relying on one monolithic AI model, I explored an architecture based on multiple specialized AI agents coordinated through LangGraph, deployed on AWS, and designed using production LLMOps principles.
The objective was not simply to build another chatbot. Instead, the goal was to design an enterprise-ready AI platform capable of:
- intelligently routing requests to domain-specific agents,
- retrieving contextual medical information using semantic search,
- validating every interaction through AI guardrails,
- delivering responses in real time using WebSockets,
- and deploying the complete solution using scalable AWS managed services.
This article walks through the architecture, design decisions, deployment strategy, and LLMOps pipeline used to build a production-oriented multi-agent healthcare AI platform.
Why Multi-Agent Architecture?
Large Language Models perform exceptionally well across a wide range of tasks. However, enterprise applications often require specialized expertise depending on the user's request.
Consider the following questions:
"Summarize the patient's chronic medical conditions."
"List the patient's upcoming appointments."
"Show current medications."
"Summarize the latest SOAP notes."
Although all four questions belong to the healthcare domain, they require completely different data sources, retrieval logic, prompts, and reasoning strategies.
Attempting to solve every problem using a single prompt quickly becomes difficult to maintain and often reduces response quality.
Instead, the application adopts a Multi-Agent Architecture, where each AI agent is responsible for a single domain of expertise.
This architecture provides several advantages:
- Improved reasoning accuracy
- Better prompt specialization
- Independent scalability
- Easier maintenance
- Reduced hallucinations
- Cleaner orchestration logic
- Domain-specific retrieval strategies
At the center of the system is a LangGraph Orchestrator responsible for deciding which specialist agent should process the user's request.
High-Level System Architecture
The application is built as a cloud-native system entirely on AWS.
Each AWS service in the architecture has a clearly defined responsibility, allowing the application to remain modular, scalable, and production ready.
Frontend Architecture
The user interface was developed using React and Vite, providing a lightweight and responsive single-page application.
Instead of traditional REST APIs, communication between the frontend and backend occurs through WebSockets.
This design choice offers several advantages.
As AI agents process requests, they continuously emit progress events such as:
- Selecting Specialist Agent
- Searching Clinical Records
- Retrieving Vector Documents
- Generating AI Response
- Validating Output
The frontend receives these events immediately, providing users with real-time feedback throughout the inference process rather than waiting for a single final response.
This significantly improves the user experience while making complex AI workflows more transparent.
CloudFront serves as the global content delivery network, ensuring low-latency delivery of frontend assets while enabling secure HTTPS communication with the backend.
FastAPI WebSocket Backend
The backend is implemented using FastAPI with native WebSocket support.
Unlike synchronous REST APIs, WebSockets maintain a persistent bidirectional connection between client and server.
Each connected session receives a unique identifier, allowing multiple users to interact with the platform simultaneously while maintaining independent conversation states.
Every incoming request follows the same execution pipeline:
- Receive user query
- Validate request
- Invoke LangGraph workflow
- Stream intermediate events
- Execute specialist agent
- Generate AI response
- Validate output
- Stream final response
Streaming intermediate events dramatically improves perceived responsiveness while enabling better debugging and observability during AI inference.
LangGraph as the Orchestration Layer
One of the most important architectural decisions was selecting LangGraph as the orchestration framework.
Traditional chains execute tasks sequentially.
Enterprise AI systems rarely operate that way.
Instead, they require branching workflows, conditional routing, shared memory, state management, retries, and independent execution nodes.
LangGraph naturally models these requirements as a graph.
The orchestrator evaluates each incoming request and determines which specialist agent should process it based on intent, context, and routing rules.
Each node within the graph performs a single well-defined responsibility.
Rather than creating one enormous prompt containing every possible instruction, each specialist agent receives only the context required for its own domain.
This approach produces cleaner prompts, better maintainability, and more reliable AI behavior.
In the next section, we'll dive into the specialized AI agents, semantic retrieval with Amazon Titan Embeddings, Amazon OpenSearch Serverless, the complete LLMOps pipeline, security guardrails, and the production deployment architecture using Amazon ECR, ECS Fargate, Application Load Balancer, and CloudFront.
Specialized AI Agents
One of the biggest limitations of traditional AI assistants is that every request is processed through the same reasoning pipeline. While this approach works for general conversations, enterprise healthcare systems require specialized expertise depending on the type of request.
To address this, the platform follows an agent-based architecture, where each AI agent is responsible for a clearly defined domain.
Instead of one large prompt trying to answer every question, the orchestration layer delegates requests to the most suitable specialist.
This design improves maintainability, enables prompt specialization, reduces hallucinations, and makes it easier to extend the platform as new business capabilities are introduced.
Orchestrator Agent
The Orchestrator acts as the decision-making layer of the system.
Every user request first reaches this agent.
Its responsibilities include:
- Understanding user intent
- Identifying the clinical domain
- Selecting the appropriate specialist agent
- Maintaining workflow state
- Coordinating downstream execution
Rather than generating responses, the orchestrator focuses exclusively on intelligent routing.
Separating orchestration from reasoning significantly improves system modularity.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# agents/orchestrator.py — hybrid routing: fast rules first, LLM fallback
async def orchestrator_node(state: AgentState) -> dict:
query = state["user_query"]
# Rule 1: Org keyword match → Knowledge Brain (instant, free)
org = _check_org_match(query)
if org:
return {"selected_agent": "knowledge", "detected_org": org}
# Rule 2: Data-question keywords → SQL Agent
if _check_sql_keywords(query):
return {"selected_agent": "sql", "detected_org": None}
# Rule 3: BMI signals (keywords or both height+weight present)
if _check_bmi(query):
return {"selected_agent": "bmi", "detected_org": None}
# Rule 4: Drug / medication keywords → Drug Agent
if _check_drug(query):
return {"selected_agent": "drug", "detected_org": None}
# Rule 5: Ambiguous — ask the LLM to classify (knowledge/sql/weather/bmi/drug/research)
prompt = ChatPromptTemplate.from_template(_CLASSIFY_PROMPT)
decision = (prompt | get_llm()).invoke({"question": query}).content.strip().lower()
selected = next(
(s for s in ["knowledge", "sql", "bmi", "drug"] if s in decision),
"research" # default fallback
)
return {"selected_agent": selected, "detected_org": None}
def route_decision(state: AgentState) -> str:
"""Conditional edge — returns next node name for add_conditional_edges."""
if state.get("guardrail_blocked"):
return "blocked"
return state.get("selected_agent", "research")Patient History Agent
The Patient History Agent specializes in longitudinal patient information.
Typical responsibilities include:
- Medical history retrieval
- Chronic disease summaries
- Previous diagnoses
- Laboratory history
- Allergies
- Past procedures
- Historical observations
Instead of searching structured databases directly, the agent retrieves semantically relevant documents through vector search before invoking the language model.
This allows responses to remain grounded in retrieved context rather than relying solely on model knowledge.
Appointment Agent
Scheduling information is fundamentally different from medical documentation.
The Appointment Agent focuses on operational healthcare workflows including:
- Upcoming appointments
- Previous consultations
- Physician schedules
- Visit summaries
- Department information
- Consultation timelines
Separating scheduling logic from clinical reasoning keeps prompts concise and domain-specific.
Pharmacy Agent
Medication-related questions require precision.
The Pharmacy Agent handles:
- Current medications
- Prescription history
- Dosage information
- Medication schedules
- Prescription summaries
- Drug administration context
By isolating pharmaceutical reasoning into its own workflow, prompts remain significantly smaller and more focused.
Clinical Notes Agent
Clinical documentation often consists of long-form physician notes containing valuable contextual information.
This agent specializes in processing:
- SOAP Notes
- Consultation summaries
- Clinical observations
- Follow-up recommendations
- Physician documentation
Large documents are retrieved semantically before summarization, allowing the language model to reason over the most relevant sections instead of the entire knowledge base.
Retrieval-Augmented Generation (RAG)
One of the primary goals of the platform was to reduce hallucinations while improving factual accuracy.
Rather than relying exclusively on foundation models, every clinical response is grounded using Retrieval-Augmented Generation (RAG).
The retrieval pipeline follows several stages:
- Convert the user query into a vector embedding.
- Perform semantic similarity search.
- Retrieve the most relevant clinical documents.
- Supply retrieved context to the foundation model.
- Generate a grounded response.
This architecture ensures that responses are based on enterprise knowledge instead of solely depending on model pretraining.
Amazon Titan Embeddings
Semantic retrieval begins with vector generation.
The application uses Amazon Titan Text Embeddings to convert natural language into dense numerical representations.
Unlike keyword search, embeddings capture semantic meaning.
For example, the following queries may retrieve similar records despite using different wording:
- "Current medications"
- "Active prescriptions"
- "Medicines being taken"
- "Patient drug history"
Because embeddings represent meaning rather than exact keywords, retrieval quality improves significantly.
Titan Embeddings also integrate seamlessly with Amazon Bedrock, simplifying enterprise deployment.
Amazon OpenSearch Serverless
After generating embeddings, vectors are stored inside Amazon OpenSearch Serverless.
Traditional relational databases perform exact matching.
Healthcare applications frequently require semantic matching instead.
OpenSearch enables Approximate Nearest Neighbor (ANN) search, allowing the system to identify documents that are conceptually similar to the user's query.
The retrieval workflow is straightforward:
Using OpenSearch Serverless eliminates infrastructure management while providing scalable vector search for production workloads.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# opensearch_client.py — semantic kNN search with optional metadata filters
def search(
self,
query_vector: list[float],
top_k: int = 5,
filter_file_type: str | None = None,
) -> list[dict]:
knn_query = {"vector": query_vector, "k": top_k}
# Optional pre-filter to restrict search to a specific file type
if filter_file_type:
query_body = {
"size": top_k,
"query": {
"bool": {
"must": [{"knn": {"embedding": knn_query}}],
"filter": [{"term": {"file_type": filter_file_type}}],
}
},
"_source": {"excludes": ["embedding"]}, # don't return raw vectors
}
else:
query_body = {
"size": top_k,
"query": {"knn": {"embedding": knn_query}},
"_source": {"excludes": ["embedding"]},
}
hits = self.client.search(index=self.index_name, body=query_body)["hits"]["hits"]
return [
{
"content": h["_source"].get("content", ""),
"score": h["_score"],
"file_name": h["_source"].get("file_name", ""),
"chunk_index": h["_source"].get("chunk_index", 0),
}
for h in hits
]Amazon Bedrock
Once contextual information has been retrieved, inference is performed through Amazon Bedrock.
Amazon Bedrock provides managed access to foundation models without requiring infrastructure provisioning or GPU management.
The platform leverages Bedrock for:
- Context-aware reasoning
- Clinical summarization
- Question answering
- Medical documentation synthesis
- Natural language generation
By combining retrieved enterprise knowledge with foundation models, responses remain grounded, relevant, and significantly more reliable.
End-to-End LLMOps Pipeline
Building production AI systems extends far beyond prompting a language model.
An effective LLMOps pipeline includes validation, retrieval, orchestration, inference, monitoring, and deployment.
The inference workflow implemented in this platform is illustrated below.
Each stage performs a single responsibility, making the overall architecture easier to monitor, debug, and scale independently.
Rather than treating inference as a single API call, the application models AI execution as an orchestrated production workflow.
In the next section, we'll explore the security architecture, AI guardrails, AWS deployment using Docker, Amazon ECR, ECS Fargate, Application Load Balancer, CloudFront, production observability, scalability considerations, and key engineering lessons learned from building the platform.
Security, Guardrails, and Production Deployment on AWS
Enterprise AI systems cannot rely solely on the capabilities of large language models. They must also address concerns such as prompt injection, malicious input, unsafe outputs, operational resilience, scalability, and observability.
This platform was designed with those production considerations in mind by combining AI orchestration with cloud-native AWS services and multiple validation layers.
AI Guardrails
One of the most important components in any production AI system is the guardrail layer.
Rather than treating validation as an afterthought, the platform performs validation before and after every model invocation.
This ensures that unsafe requests are filtered before reaching the foundation model and that generated responses satisfy predefined safety requirements before being returned to the user.
The validation workflow consists of two independent stages.
1
2
3
4
5
6
7
8
9
10
11
12
13
User Query
│
▼
Input Guardrails
│
▼
Foundation Model
│
▼
Output Guardrails
│
▼
Validated Response
Separating these responsibilities provides defense in depth while reducing the risk of unsafe model behavior.
Input Validation
Every incoming request passes through an input validation pipeline before reaching the orchestration layer.
The objective is to ensure that the language model only processes legitimate requests.
The validation layer performs several checks, including:
- Prompt injection detection
- Input sanitization
- Invalid character filtering
- Malicious prompt detection
- Request normalization
- Empty or malformed request validation
Examples include attempts to override system instructions, expose hidden prompts, manipulate routing behavior, or inject malicious content.
By filtering these requests early, unnecessary inference costs are avoided while improving the overall security posture of the application.
Output Validation
Model responses also undergo validation before being streamed back to the client.
Unlike traditional applications, AI systems may occasionally generate responses that do not comply with organizational requirements.
The output validation layer ensures that generated content satisfies predefined quality and safety rules.
Typical validation includes:
- Response completeness
- Safe content verification
- Structured output validation
- Hallucination reduction through retrieved context
- Formatting consistency
- Response sanitization
This additional verification step improves response reliability while creating a more predictable user experience.
Why Guardrails Matter
Large Language Models are powerful reasoning engines, but they are probabilistic systems.
Enterprise applications therefore require deterministic validation layers around the model.
Instead of assuming every generated response is correct, guardrails establish clear boundaries regarding what the model is allowed to process and what can ultimately be returned to the end user.
This layered approach aligns with modern LLMOps practices where AI systems are treated as production software rather than standalone models.
Containerization with Docker
To ensure portability and reproducibility, the backend application is packaged as a Docker container.
Containerization offers several operational benefits:
- Consistent runtime environments
- Dependency isolation
- Simplified deployment
- Predictable scaling
- Infrastructure portability
Every application dependency, including FastAPI, LangGraph, Python libraries, and runtime configurations, is packaged into a single deployable image.
This guarantees that development, testing, and production environments remain consistent.
Amazon Elastic Container Registry (Amazon ECR)
Once the container image is built, it is stored in Amazon Elastic Container Registry (Amazon ECR).
ECR serves as the centralized repository for application images before deployment.
Using ECR provides:
- Secure image storage
- Version management
- Integration with Amazon ECS
- IAM-based access control
- High availability
Each deployment begins by building a new Docker image, pushing it to ECR, and updating the ECS service.
Amazon ECS Fargate
Instead of provisioning and maintaining EC2 instances, the application runs on Amazon ECS Fargate.
Fargate eliminates server management while allowing developers to focus entirely on application logic.
Key advantages include:
- Serverless container execution
- Automatic infrastructure management
- Elastic scaling
- Integrated networking
- IAM integration
- High availability
Because the platform is stateless, multiple containers can run simultaneously behind a load balancer without requiring changes to application logic.
This makes horizontal scaling straightforward as user traffic increases.
Application Load Balancer
AI conversations rely on persistent WebSocket connections.
The Application Load Balancer serves as the entry point for all incoming traffic and performs several critical responsibilities.
These include:
- HTTPS termination
- WebSocket upgrades
- Request routing
- Health checks
- Traffic distribution
- High availability
The load balancer distributes requests across multiple ECS tasks, allowing the application to continue operating even if individual containers become unavailable.
Amazon CloudFront
The frontend application is delivered globally through Amazon CloudFront.
Using CloudFront provides several advantages beyond content caching.
Benefits include:
- Global edge locations
- Lower latency
- HTTPS delivery
- Improved frontend performance
- Reduced origin traffic
- Better user experience
CloudFront ensures that users access frontend assets from the nearest edge location while securely forwarding requests to backend services.
Observability and Monitoring
Observability is a fundamental aspect of production AI systems.
Understanding how requests move through the orchestration pipeline is essential for debugging, optimization, and operational reliability.
The application integrates with Amazon CloudWatch for centralized logging and monitoring.
Key operational metrics include:
- Request latency
- Agent selection
- Retrieval duration
- Inference time
- Container health
- Application logs
- Runtime exceptions
These metrics provide valuable insights into system performance while simplifying production troubleshooting.
Scalability
The architecture was intentionally designed using stateless services to enable horizontal scaling.
As demand increases, additional ECS tasks can be launched behind the Application Load Balancer without modifying application logic.
The architecture supports independent scaling across multiple layers.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
CloudFront
↓
Application Load Balancer
↓
Multiple ECS Tasks
↓
LangGraph Execution
↓
Amazon Bedrock
↓
Amazon OpenSearch Serverless
Because each service is independently managed, the platform can scale based on workload characteristics while maintaining operational efficiency.
Engineering Decisions
Several architectural decisions significantly influenced the final design.
Why LangGraph?
Instead of traditional sequential chains, LangGraph enables stateful orchestration, conditional routing, and specialized workflows.
This makes it well suited for enterprise multi-agent systems.
Why Amazon Bedrock?
Amazon Bedrock removes the operational complexity of managing foundation models while providing secure access to enterprise-grade AI capabilities.
Developers can focus on application logic rather than infrastructure management.
Why Amazon Titan Embeddings?
Embedding quality directly influences retrieval quality.
Amazon Titan Embeddings provide semantic representations optimized for enterprise search workloads while integrating naturally with Amazon Bedrock.
Why OpenSearch Serverless?
Vector databases are essential for Retrieval-Augmented Generation.
Amazon OpenSearch Serverless provides scalable semantic retrieval without infrastructure administration.
Why WebSockets?
AI inference is an iterative process.
Streaming intermediate events dramatically improves user experience by exposing workflow progress rather than forcing users to wait for a single final response.
Lessons Learned
Building production AI systems extends far beyond integrating a language model.
Several key lessons emerged throughout development:
- AI systems should be composed of specialized components rather than monolithic prompts.
- Retrieval quality is often more important than prompt complexity.
- Guardrails are essential for enterprise deployments.
- Observability should be incorporated from the beginning rather than added later.
- Stateless architectures simplify scaling and operational management.
- Cloud-native managed services significantly reduce infrastructure complexity while improving reliability.
These principles helped transform a prototype into a production-oriented multi-agent AI platform built using modern AWS services.
One production challenge worth highlighting was the research agent's web search integration. The initial implementation used Google Custom Search Engine (CSE), which worked in local development but returned 403 errors in the deployed ECS environment due to an organisation-level GCP policy restriction that blocked API access from external infrastructure. Rather than spending time on GCP policy investigation, the decision was made to switch to Tavily — a search API built specifically for LLM applications. The migration took under an hour and the resulting integration was simpler and more reliable. This experience reinforced an important principle: when an external dependency fails at the infrastructure boundary, evaluate whether a purpose-built alternative exists before attempting to debug policy-level access restrictions.
Evaluation and Observability
Building a production-grade AI system goes beyond response generation. Continuous evaluation and observability are essential to ensure response quality, retrieval effectiveness, and system reliability over time.
The platform is designed with an extensible LLMOps pipeline that supports evaluation alongside operational monitoring. While runtime observability is achieved through structured trace logging, PostgreSQL persistence, and Amazon CloudWatch, the architecture can be extended with automated evaluation workflows to continuously assess retrieval and response quality.
Key evaluation dimensions include:
- Answer Relevance – Measures how effectively the generated response addresses the user's query.
- Context Relevance – Evaluates whether the retrieved documents are relevant to the user's request.
- Groundedness (Faithfulness) – Ensures that responses remain grounded in the retrieved context and minimize hallucinations.
- Response Quality – Assesses the completeness, coherence, and usefulness of generated responses.
- Latency Metrics – Tracks retrieval, orchestration, inference, and end-to-end response times for performance optimization.
By combining evaluation metrics with operational telemetry collected through Amazon CloudWatch and structured execution traces, the platform is designed to support continuous improvement of retrieval quality, prompt engineering, and overall system performance as part of an enterprise LLMOps lifecycle.
Frontend Experience
While the backend focuses on orchestration, retrieval, and inference, the frontend was designed to provide a responsive and transparent AI experience.The interface communicates with the backend using secure WebSockets, allowing users to observe each stage of the AI workflow in real time—from request validation and agent selection to semantic retrieval and response generation.The screenshots below illustrate the production interface.
Engineering Lessons Learned
Although building the multi-agent architecture was technically rewarding, the most valuable lessons came from solving production deployment challenges rather than writing prompts.
One of the biggest challenges was enabling secure WebSocket communication after deployment. While the application functioned correctly in the local environment, the browser blocked WebSocket connections once the frontend was served over HTTPS because the backend was still exposed through an unsecured endpoint. Migrating the networking layer to Amazon CloudFront with secure WebSocket support resolved the issue while improving the overall production architecture.
Another important lesson involved containerized deployments. During testing on Amazon ECS, the application failed because a required Amazon Bedrock model configuration was missing from the runtime environment, even though everything worked locally. This highlighted the importance of validating environment variables and runtime configuration independently of local development.
Finally, designing the LangGraph orchestration reinforced the value of specialized AI agents. Separating responsibilities into dedicated agents for appointments, patient history, pharmacy information, and clinical documentation resulted in a more modular architecture, simpler prompts, and a cleaner orchestration workflow than a single monolithic agent.
Architectural Design Decisions
Why This Architecture?
While designing the platform, I evaluated several architectural approaches before settling on a LangGraph-based multi-agent workflow.A single-agent architecture was initially considered because it simplifies orchestration. However, as healthcare capabilities expanded across appointments, patient history, pharmacy information, and clinical documentation, prompt complexity increased significantly and became difficult to maintain.Separating responsibilities into specialized agents resulted in cleaner prompts, improved modularity, and simpler reasoning while allowing each component to evolve independently.Similarly, OpenSearch Serverless was selected over traditional keyword search to enable semantic retrieval using Titan Embeddings, while Amazon ECS Fargate removed the operational overhead of managing container infrastructure.These decisions were driven not only by functionality, but also by long-term maintainability, scalability, and operational simplicity.
Conclusion
Enterprise AI applications require far more than simply connecting a large language model to a frontend.
A production-ready platform combines intelligent orchestration, semantic retrieval, cloud-native infrastructure, validation layers, security controls, observability, and scalable deployment patterns.
By integrating LangGraph for orchestration, Amazon Bedrock for foundation models, Amazon Titan Embeddings for semantic understanding, Amazon OpenSearch Serverless for vector retrieval, FastAPI WebSockets for real-time communication, and AWS services such as Amazon ECS Fargate, Amazon ECR, Application Load Balancer, CloudFront, and CloudWatch, it is possible to build a modern LLMOps platform that is modular, scalable, and enterprise ready.
As generative AI continues to evolve, the future of enterprise applications will increasingly depend on specialized AI agents working together through orchestrated workflows rather than isolated models. Cloud-native architectures, robust retrieval mechanisms, and comprehensive validation layers will play a central role in building reliable, secure, and production-grade AI systems on AWS.
# agentic-ai# generative-ai# amazon-bedrock-agents# amazon-opensearch# amazon-elastic-container-registry
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article