
Building a Vendor Evaluation Assistant: A 5-Agent Agentic AI System on AWS
A 5-agent AI system built with Strands SDK and Amazon Bedrock Nova Pro that automates enterprise vendor evaluation — researching, scoring, risk-assessing, and reporting on vendors end-to-end via a FastAPI REST API.
Building a Vendor Evaluation Assistant: A 5-Agent Agentic AI System on AWS
By Sweta Jha | AWS Community Builder (AI/ML)
Introduction
Vendor procurement is one of the most consequential decisions an enterprise makes. Yet the process remains painfully manual — spreadsheets, one-off reviews, inconsistent scoring rubrics, and no audit trail.
What if five AI agents could coordinate to research, score, risk-assess, and report on vendors end-to-end?
In this post, I walk through the full architecture and production code for a Vendor Evaluation Assistant — a multi-agent system built with the Strands Agents SDK, Amazon Bedrock Nova Pro, and AWS AgentCore Runtime. This is real, runnable code designed for the AWS Community Builder audience.
System Architecture
The system is composed of five specialized agents, orchestrated by a central coordinator:
| Agent | Model | Role |
|---|---|---|
| OrchestratorAgent | Nova Pro | Routes tasks, synthesizes all outputs, returns final result |
| ResearchAgent | Nova Pro | Gathers vendor intelligence: reviews, pricing, certifications |
| ScoringAgent | Nova Pro | Applies weighted MCDA scoring matrix across all vendors |
| RiskAgent | Nova Pro | Assesses compliance (HIPAA, SOC2, GDPR), financial, and exit risk |
| ReportAgent | Nova Pro | Generates full Markdown evaluation report with score tables |
Workflow Diagram
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
User Request (POST /evaluate)
│
▼
┌─────────────────────────────────┐
│ OrchestratorAgent │
│ Coordinates · Synthesizes │
└────┬──────┬──────┬──────┬───────┘
│ │ │ │
▼ ▼ ▼ ▼
Research Scoring Risk Report
Agent Agent Agent Agent
│ │ │ │
└──────┴──────┴──────┘
│
▼
┌──────────────────────┐
│ Evaluation Report │
│ REST /evaluate resp │
└──────────────────────┘
AWS Infrastructure:
Amazon Bedrock (Nova Pro · us-east-1)
AgentCore Runtime · Strands SDK
FastAPI + S3 · REST APITechnology Stack
| Layer | Technology |
|---|---|
| Agent Framework | Strands Agents SDK (AWS Open Source) |
| LLM | Amazon Bedrock — Nova Pro (us.amazon.nova-pro-v1:0) |
| Runtime | AWS AgentCore Runtime |
| API Layer | FastAPI + Uvicorn |
| Validation | Pydantic v2 |
| Cloud | AWS (S3, Lambda-ready, ECS-deployable) |
Key Design Decisions
1. Strands Agents SDK
Strands SDK from AWS makes it trivial to define agents with tools using Python decorators. Each specialist agent is an
Agent instance with a focused system prompt and a small set of @tool-decorated functions.1
2
3
4
5
6
7
from strands import Agent, tool
from strands.models import BedrockModel
def fetch_vendor_pricing(vendor_name: str) -> dict:
"""Fetches pricing structure for a vendor."""
...2. Amazon Bedrock Nova Pro
All five agents use
us.amazon.nova-pro-v1:0 — Amazon's frontier model with strong instruction-following, tool use, and JSON output. The Orchestrator uses temperature=0.1 for creative synthesis; scoring and risk agents use 0.0 for deterministic, reproducible results.3. Multi-Criteria Decision Analysis (MCDA)
The ScoringAgent implements a weighted MCDA model. Each criterion (features, security, support, value, integration) receives a weight from the user. Scores are normalized 0–10 and a composite weighted score is computed per vendor — producing an objective, auditable ranking.
4. 4-Dimension Risk Framework
The RiskAgent evaluates vendors across four risk dimensions:
- Compliance Risk — HIPAA, SOC2, ISO27001, GDPR certification gaps
- Financial Risk — funding runway, burn rate, acquisition probability
- Exit Risk — data portability, contract lock-in, migration complexity
- Operational Risk — SLA track record, support quality, single points of failure
5. FastAPI REST Layer
The entire pipeline is exposed via a FastAPI endpoint (
POST /evaluate), making it trivially deployable as an AWS Lambda function, ECS container, or EC2 service.Pipeline Workflow
End-to-end request flow:
- User sends
POST /evaluatewith vendor list, criteria weights, and business context - OrchestratorAgent parses the request and dispatches ResearchAgent per vendor
- ResearchAgent calls
@toolfunctions:search_vendor_reviews,fetch_vendor_pricing,check_certifications - Research outputs are sent to ScoringAgent, which computes MCDA scores and vendor rankings
- Scored vendors are passed to RiskAgent for compliance, financial, and exit risk assessment
- All data is aggregated and sent to ReportAgent, which generates a structured Markdown report
- OrchestratorAgent synthesizes everything with an LLM prompt and returns the final JSON response
Code Walkthrough
Orchestrator Agent (agents/orchestrator.py)
The Orchestrator is the brain of the system. It holds references to all specialist agents and exposes
@tool-decorated dispatch methods so that the Bedrock LLM can reason about when to call which agent.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
ORCHESTRATOR_SYSTEM_PROMPT = """You are the Vendor Evaluation Orchestrator for enterprise procurement decisions.
Your role is to coordinate specialist AI agents to evaluate vendors across multiple dimensions.
Workflow:
1. Parse vendor evaluation request and extract vendor list + evaluation criteria
2. Dispatch ResearchAgent to gather vendor intelligence per vendor
3. Send research data to ScoringAgent for quantitative scoring
4. Route scored vendors to RiskAgent for compliance + risk assessment
5. Aggregate all outputs and send to ReportAgent for final report generation
6. Return structured evaluation report to the user"""
class OrchestratorAgent:
def __init__(self):
self.model = BedrockModel(
model_id="us.amazon.nova-pro-v1:0",
region_name="us-east-1",
temperature=0.1,
)
self.agent = Agent(
model=self.model,
system_prompt=ORCHESTRATOR_SYSTEM_PROMPT,
tools=[self._dispatch_research, self._dispatch_scoring,
self._dispatch_risk, self._dispatch_report],
)
self.research_agent = ResearchAgent()
self.scoring_agent = ScoringAgent()
self.risk_agent = RiskAgent()
self.report_agent = ReportAgent()
def _dispatch_research(self, vendors: list[str], criteria: dict) -> dict:
"""Dispatch ResearchAgent for each vendor."""
results = {}
for vendor in vendors:
results[vendor] = self.research_agent.run(vendor, criteria)
return results
def evaluate(self, request: dict) -> dict:
vendors = request["vendors"]
criteria = request.get("criteria", {})
weights = request.get("weights", {})
# Step 1: Research
research_data = self._dispatch_research(vendors, criteria)
# Step 2: Scoring
scored_vendors = self._dispatch_scoring(research_data, weights)
# Step 3: Risk Assessment
risk_data = self._dispatch_risk(scored_vendors)
# Step 4: Report Generation
report_path = self._dispatch_report(
{"research": research_data, "scores": scored_vendors, "risks": risk_data},
{"requester": request.get("requester")}
)
return {
"status": "completed",
"recommended_vendor": scored_vendors.get("top_vendor"),
"report_path": report_path,
"scores": scored_vendors,
"risks": risk_data,
}Research Agent (agents/research_agent.py)
ResearchAgent uses three
@tool functions to gather vendor data. In production, these connect to G2 API, Crunchbase, or web scraping pipelines.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
RESEARCH_SYSTEM_PROMPT = """You are a Vendor Research Specialist AI agent.
For each vendor, research and return:
- Company overview (founded, HQ, employee count, funding stage)
- Product features and capabilities relevant to the evaluation criteria
- Pricing model (subscription, usage-based, enterprise licensing)
- Customer reviews and satisfaction scores (G2, Gartner Peer Insights)
- Market presence (Fortune 500 customers, industry certifications)
- Recent news (acquisitions, product launches, layoffs, security incidents)"""
def search_vendor_reviews(vendor_name: str, category: str) -> dict:
"""Fetches vendor reviews from G2/Gartner. Replace with real API calls in production."""
return {
"overall_rating": 4.2,
"total_reviews": 847,
"ease_of_use": 4.1,
"customer_support": 3.9,
"value_for_money": 3.8,
"feature_rating": 4.3,
"recommended_pct": 82,
"top_pros": ["Easy integration", "Strong API", "Good documentation"],
"top_cons": ["Steep learning curve", "Pricing not transparent"],
}
def fetch_vendor_pricing(vendor_name: str) -> dict:
"""Fetches vendor pricing. Replace with vendor API or pricing intelligence tool."""
return {
"model": "Per-seat SaaS subscription",
"starter_tier": "$49/user/month (up to 10 users)",
"enterprise_tier": "Custom pricing (100+ users)",
"annual_discount": "20% discount for annual commitment",
"implementation_fee": "$5,000 - $25,000 (one-time)",
}
def check_certifications(vendor_name: str) -> dict:
"""Checks vendor compliance certifications."""
return {
"hipaa_compliant": True,
"soc2_type2": True,
"iso_27001": True,
"gdpr_compliant": True,
"fedramp": False,
"baa_available": True,
"last_audit_date": "2024-Q3",
"data_residency_options": ["US-East", "US-West", "EU-Frankfurt"],
}
class ResearchAgent:
def __init__(self):
self.model = BedrockModel(
model_id="us.amazon.nova-pro-v1:0",
region_name="us-east-1",
temperature=0.1,
)
self.agent = Agent(
model=self.model,
system_prompt=RESEARCH_SYSTEM_PROMPT,
tools=[search_vendor_reviews, fetch_vendor_pricing, check_certifications],
)
def run(self, vendor_name: str, criteria: dict) -> dict:
prompt = f"""
Research the vendor "{vendor_name}" for enterprise procurement evaluation.
Focus areas: {json.dumps(criteria, indent=2)}
Use all available tools, then return a structured JSON vendor profile.
"""
response = self.agent(prompt)
return {"vendor_name": vendor_name, "raw_response": str(response)}Scoring Agent (agents/scoring_agent.py)
ScoringAgent applies the weighted MCDA model.
compute_weighted_score normalizes scores 0–10 and returns a composite with a letter grade; rank_vendors sorts by composite; identify_score_gaps reveals where vendors diverge most.1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
def compute_weighted_score(vendor_scores: dict, weights: dict) -> dict:
"""Computes weighted composite score for a vendor."""
total_weight = sum(weights.values())
normalized_weights = {k: v / total_weight for k, v in weights.items()}
weighted_scores = {}
composite = 0.0
for criterion, raw_score in vendor_scores.items():
weight = normalized_weights.get(criterion, 0)
weighted = raw_score * weight
weighted_scores[criterion] = {
"raw_score": raw_score,
"weight": weight,
"weighted_score": round(weighted, 3),
}
composite += weighted
return {
"weighted_scores": weighted_scores,
"composite_score": round(composite, 3),
"grade": _score_to_grade(composite),
}
def _score_to_grade(score: float) -> str:
if score >= 8.5: return "A+"
if score >= 8.0: return "A"
if score >= 7.5: return "A-"
if score >= 7.0: return "B+"
if score >= 6.5: return "B"
return "C"
def rank_vendors(vendor_composites: dict) -> list:
"""Ranks vendors by composite score."""
ranked = sorted(vendor_composites.items(), key=lambda x: x[1], reverse=True)
return [{"rank": i+1, "vendor": v, "score": s} for i, (v, s) in enumerate(ranked)]
def identify_score_gaps(vendor_scores_matrix: dict) -> dict:
"""Identifies largest score gaps between vendors per criterion."""
criteria = set()
for scores in vendor_scores_matrix.values():
criteria.update(scores.keys())
gaps = {}
for criterion in criteria:
scores_for_criterion = {v: s.get(criterion, 0) for v, s in vendor_scores_matrix.items()}
max_vendor = max(scores_for_criterion, key=scores_for_criterion.get)
min_vendor = min(scores_for_criterion, key=scores_for_criterion.get)
gaps[criterion] = {
"best_vendor": max_vendor,
"best_score": scores_for_criterion[max_vendor],
"worst_vendor": min_vendor,
"worst_score": scores_for_criterion[min_vendor],
"gap": round(scores_for_criterion[max_vendor] - scores_for_criterion[min_vendor], 2),
}
return gapsRisk Agent (agents/risk_agent.py)
RiskAgent assesses three risk dimensions and computes a composite risk score with a severity label (LOW / MEDIUM / HIGH / CRITICAL).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
def assess_compliance_risk(vendor_name: str, certifications: dict, industry: str = "healthcare") -> dict:
"""Assesses compliance risk based on certifications and industry requirements."""
risk_score = 0
flags = []
industry_requirements = {
"healthcare": ["hipaa_compliant", "soc2_type2", "baa_available"],
"finance": ["soc2_type2", "pci_dss", "iso_27001"],
"general": ["soc2_type2", "gdpr_compliant"],
}
required = industry_requirements.get(industry, industry_requirements["general"])
for req in required:
if not certifications.get(req, False):
flags.append(f"Missing required certification: {req}")
risk_score += 3
return {
"compliance_risk_score": min(10, risk_score),
"compliance_flags": flags,
"audit_currency": certifications.get("last_audit_date", "unknown"),
}
def assess_financial_risk(vendor_name: str) -> dict:
"""Financial risk: runway, burn rate, acquisition risk. Connect to Crunchbase/D&B in production."""
mock_financial = {
"funding_stage": "Series C",
"burn_rate_runway_months": 24,
"acquisition_risk": "Low — well-capitalized, independent",
}
risk_score = 2
flags = []
if mock_financial["burn_rate_runway_months"] < 12:
risk_score += 4
flags.append("Low runway (<12 months) — financial instability risk")
return {"financial_risk_score": min(10, risk_score), "financial_flags": flags}
def assess_exit_risk(vendor_name: str, pricing: dict) -> dict:
"""Data lock-in and exit risk assessment."""
risk_score = 0
flags = []
mitigations = []
has_data_export = True
min_contract_months = 12
if not has_data_export:
risk_score += 4
flags.append("No data export capability — high lock-in risk")
else:
mitigations.append("Data export API available")
if min_contract_months >= 24:
risk_score += 3
flags.append(f"Long minimum contract ({min_contract_months} months)")
return {
"exit_risk_score": min(10, risk_score),
"exit_flags": flags,
"exit_mitigations": mitigations,
}Report Agent (agents/report_agent.py)
ReportAgent generates a full Markdown report with score tables, risk summaries, rankings, and an implementation roadmap.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
def format_score_table(vendor_scores: dict) -> str:
"""Formats vendor scores as a Markdown comparison table."""
vendors = list(vendor_scores.keys())
all_criteria = sorted(set(
criterion
for scores in vendor_scores.values()
for criterion in scores.get("weighted_scores", {}).keys()
))
header = "| Criterion | " + " | ".join(vendors) + " |"
separator = "|-----------|" + "|".join(["--------"] * len(vendors)) + "|"
rows = []
for criterion in all_criteria:
row = f"| {criterion.replace('_', ' ').title()} |"
for vendor in vendors:
score = vendor_scores[vendor].get("weighted_scores", {}).get(criterion, {}).get("raw_score", "N/A")
row += f" {score} |"
rows.append(row)
composite_row = "| **COMPOSITE** |"
for vendor in vendors:
comp = vendor_scores[vendor].get("composite_score", "N/A")
grade = vendor_scores[vendor].get("grade", "")
composite_row += f" **{comp} ({grade})** |"
rows.append(composite_row)
return "\n".join([header, separator] + rows)FastAPI Layer (api/main.py)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
from fastapi import FastAPI
from pydantic import BaseModel
from agents.orchestrator import OrchestratorAgent
app = FastAPI(
title="Vendor Evaluation Assistant",
description="Multi-Agent AI System for Enterprise Vendor Procurement Evaluation",
version="1.0.0",
)
orchestrator = OrchestratorAgent()
class VendorEvaluationRequest(BaseModel):
vendors: list[str]
criteria: dict[str, float] = {
"features": 0.3, "security": 0.25, "support": 0.2,
"value": 0.15, "integration": 0.1
}
context: str = "Enterprise software procurement"
requester: str = "Procurement Team"
industry: str = "general"
async def evaluate_vendors(request: VendorEvaluationRequest):
"""
Trigger the full 5-agent vendor evaluation pipeline.
Returns composite scores, risk assessment, rankings, and report path.
"""
result = orchestrator.evaluate(request.model_dump())
return result
async def agent_status():
return {
"pipeline": [
{"agent": "OrchestratorAgent", "role": "Coordinates all agents"},
{"agent": "ResearchAgent", "role": "Vendor intelligence gathering"},
{"agent": "ScoringAgent", "role": "Weighted MCDA scoring"},
{"agent": "RiskAgent", "role": "Compliance & risk assessment"},
{"agent": "ReportAgent", "role": "Report generation"},
],
"model": "us.amazon.nova-pro-v1:0",
"runtime": "AWS AgentCore Runtime",
}Running the Project
Prerequisites
- AWS account with Bedrock Nova Pro access enabled in
us-east-1 - Python 3.11+
- IAM role with
bedrock:InvokeModelpermissions
Setup
1
2
3
4
5
git clone https://github.com/your-repo/vendor-eval-assistant
cd vendor-eval-assistant
pip install -r requirements.txt
export AWS_DEFAULT_REGION=us-east-1
uvicorn api.main:app --reload --port 8000Sample API Call
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
curl -X POST http://localhost:8000/evaluate \
-H "Content-Type: application/json" \
-d '{
"vendors": ["Salesforce", "HubSpot", "Microsoft Dynamics"],
"criteria": {
"features": 0.30,
"security": 0.25,
"support": 0.20,
"value": 0.15,
"integration": 0.10
},
"context": "CRM for 500-person enterprise sales team",
"requester": "Procurement Team",
"industry": "general"
}'Sample Response
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
"status": "completed",
"recommended_vendor": "Salesforce",
"report_path": "outputs/vendor_evaluation_20250625_143022.md",
"scores": {
"rankings": [
{ "rank": 1, "vendor": "Salesforce", "score": 7.84 },
{ "rank": 2, "vendor": "HubSpot", "score": 7.41 },
{ "rank": 3, "vendor": "Microsoft Dynamics", "score": 6.93 }
],
"top_vendor": "Salesforce"
},
"risks": {
"vendor_risks": {
"Salesforce": { "risk_level": "LOW", "composite_risk_score": 2.1 },
"HubSpot": { "risk_level": "LOW", "composite_risk_score": 2.4 },
"Microsoft Dynamics": {
"risk_level": "MEDIUM",
"composite_risk_score": 3.8
}
}
}
}File Structure
1
2
3
4
5
6
7
8
9
10
11
vendor-eval-assistant/
├── agents/
│ ├── __init__.py
│ ├── orchestrator.py ← Central coordinator + synthesis
│ ├── research_agent.py ← Reviews, pricing, certifications
│ ├── scoring_agent.py ← Weighted MCDA scoring matrix
│ ├── risk_agent.py ← HIPAA/SOC2/financial/exit risk
│ └── report_agent.py ← Markdown report generation
├── api/
│ └── main.py ← FastAPI REST layer
└── requirements.txtProduction Considerations
- Replace mock
@toolfunctions with real API integrations (G2, Crunchbase, Clearbit) - Add async agent dispatch for parallel vendor research across vendors
- Integrate Amazon Bedrock Knowledge Bases for vendor history and past contracts
- Add a Human Approval Tool before report finalization for high-stakes decisions
- Deploy with AgentCore Runtime for managed scaling and observability
- Store evaluation history in DynamoDB for audit trail and trend analysis
- Add streaming responses via FastAPI
StreamingResponsefor long-running evaluations
Why This Pattern Works
The Vendor Evaluation Assistant demonstrates a clean pattern for production agentic AI:
- Focused Orchestrator — coordinates without doing domain work itself
- Stateless specialist agents — each agent owns one concern, makes it testable and swappable
@tool-decorated functions — explicit, typed, and easy to mock in tests- REST API layer — decouples the agent system from any frontend or consumer
Each agent does exactly one thing well. You can swap the ScoringAgent for a more sophisticated model, add a NegotiationAgent, or plug in a different LLM — without touching the rest of the pipeline.
This is the kind of system that saves enterprise procurement teams hundreds of hours per year.
About the Author
Sweta Jha is a Senior Data Scientist and AI/ML Engineer at Carelon Global Solutions (Elevance Health). AWS Community Builder in AI/ML. Google Cloud Agentic AI Hackathon winner 2025. Specializes in GenAI, LLM orchestration, RAG pipelines, Agentic AI, and MLOps across healthcare, BFSI, and enterprise domains.
- GitHub: https://github.com/its-me-sweta/Vendor_Evaluation_AI_Assistant.git
Built on AWS | Strands Agents SDK | Amazon Bedrock Nova Pro | FastAPI
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article