AWS Builder Center
Building a Vendor Evaluation Assistant: A 5-Agent Agentic AI System on AWS

Building a Vendor Evaluation Assistant: A 5-Agent Agentic AI System on AWS

A 5-agent AI system built with Strands SDK and Amazon Bedrock Nova Pro that automates enterprise vendor evaluation — researching, scoring, risk-assessing, and reporting on vendors end-to-end via a FastAPI REST API.

Building a Vendor Evaluation Assistant: A 5-Agent Agentic AI System on AWS

By Sweta Jha | AWS Community Builder (AI/ML)

Introduction

Vendor procurement is one of the most consequential decisions an enterprise makes. Yet the process remains painfully manual — spreadsheets, one-off reviews, inconsistent scoring rubrics, and no audit trail.
What if five AI agents could coordinate to research, score, risk-assess, and report on vendors end-to-end?
In this post, I walk through the full architecture and production code for a Vendor Evaluation Assistant — a multi-agent system built with the Strands Agents SDK, Amazon Bedrock Nova Pro, and AWS AgentCore Runtime. This is real, runnable code designed for the AWS Community Builder audience.

System Architecture

The system is composed of five specialized agents, orchestrated by a central coordinator:
AgentModelRole
OrchestratorAgentNova ProRoutes tasks, synthesizes all outputs, returns final result
ResearchAgentNova ProGathers vendor intelligence: reviews, pricing, certifications
ScoringAgentNova ProApplies weighted MCDA scoring matrix across all vendors
RiskAgentNova ProAssesses compliance (HIPAA, SOC2, GDPR), financial, and exit risk
ReportAgentNova ProGenerates full Markdown evaluation report with score tables

Workflow Diagram

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
User Request (POST /evaluate)
│
▼
┌─────────────────────────────────┐
│ OrchestratorAgent │
│ Coordinates · Synthesizes │
└────┬──────┬──────┬──────┬───────┘
│ │ │ │
▼ ▼ ▼ ▼
Research Scoring Risk Report
Agent Agent Agent Agent
│ │ │ │
└──────┴──────┴──────┘
│
▼
┌──────────────────────┐
│ Evaluation Report │
│ REST /evaluate resp │
└──────────────────────┘

AWS Infrastructure:
Amazon Bedrock (Nova Pro · us-east-1)
AgentCore Runtime · Strands SDK
FastAPI + S3 · REST API

Technology Stack

LayerTechnology
Agent FrameworkStrands Agents SDK (AWS Open Source)
LLMAmazon Bedrock — Nova Pro (us.amazon.nova-pro-v1:0)
RuntimeAWS AgentCore Runtime
API LayerFastAPI + Uvicorn
ValidationPydantic v2
CloudAWS (S3, Lambda-ready, ECS-deployable)

Key Design Decisions

1. Strands Agents SDK

Strands SDK from AWS makes it trivial to define agents with tools using Python decorators. Each specialist agent is an Agent instance with a focused system prompt and a small set of @tool-decorated functions.
1
2
3
4
5
6
7
from strands import Agent, tool
from strands.models import BedrockModel

@tool
def fetch_vendor_pricing(vendor_name: str) -> dict:
"""Fetches pricing structure for a vendor."""
...

2. Amazon Bedrock Nova Pro

All five agents use us.amazon.nova-pro-v1:0 — Amazon's frontier model with strong instruction-following, tool use, and JSON output. The Orchestrator uses temperature=0.1 for creative synthesis; scoring and risk agents use 0.0 for deterministic, reproducible results.

3. Multi-Criteria Decision Analysis (MCDA)

The ScoringAgent implements a weighted MCDA model. Each criterion (features, security, support, value, integration) receives a weight from the user. Scores are normalized 0–10 and a composite weighted score is computed per vendor — producing an objective, auditable ranking.

4. 4-Dimension Risk Framework

The RiskAgent evaluates vendors across four risk dimensions:
  • Compliance Risk — HIPAA, SOC2, ISO27001, GDPR certification gaps
  • Financial Risk — funding runway, burn rate, acquisition probability
  • Exit Risk — data portability, contract lock-in, migration complexity
  • Operational Risk — SLA track record, support quality, single points of failure

5. FastAPI REST Layer

The entire pipeline is exposed via a FastAPI endpoint (POST /evaluate), making it trivially deployable as an AWS Lambda function, ECS container, or EC2 service.

Pipeline Workflow

End-to-end request flow:
  1. User sends POST /evaluate with vendor list, criteria weights, and business context
  2. OrchestratorAgent parses the request and dispatches ResearchAgent per vendor
  3. ResearchAgent calls @tool functions: search_vendor_reviews, fetch_vendor_pricing, check_certifications
  4. Research outputs are sent to ScoringAgent, which computes MCDA scores and vendor rankings
  5. Scored vendors are passed to RiskAgent for compliance, financial, and exit risk assessment
  6. All data is aggregated and sent to ReportAgent, which generates a structured Markdown report
  7. OrchestratorAgent synthesizes everything with an LLM prompt and returns the final JSON response

Code Walkthrough

Orchestrator Agent (agents/orchestrator.py)

The Orchestrator is the brain of the system. It holds references to all specialist agents and exposes @tool-decorated dispatch methods so that the Bedrock LLM can reason about when to call which agent.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
ORCHESTRATOR_SYSTEM_PROMPT = """You are the Vendor Evaluation Orchestrator for enterprise procurement decisions.
Your role is to coordinate specialist AI agents to evaluate vendors across multiple dimensions.

Workflow:
1. Parse vendor evaluation request and extract vendor list + evaluation criteria
2. Dispatch ResearchAgent to gather vendor intelligence per vendor
3. Send research data to ScoringAgent for quantitative scoring
4. Route scored vendors to RiskAgent for compliance + risk assessment
5. Aggregate all outputs and send to ReportAgent for final report generation
6. Return structured evaluation report to the user"""


class OrchestratorAgent:
def __init__(self):
self.model = BedrockModel(
model_id="us.amazon.nova-pro-v1:0",
region_name="us-east-1",
temperature=0.1,
)
self.agent = Agent(
model=self.model,
system_prompt=ORCHESTRATOR_SYSTEM_PROMPT,
tools=[self._dispatch_research, self._dispatch_scoring,
self._dispatch_risk, self._dispatch_report],
)
self.research_agent = ResearchAgent()
self.scoring_agent = ScoringAgent()
self.risk_agent = RiskAgent()
self.report_agent = ReportAgent()

def _dispatch_research(self, vendors: list[str], criteria: dict) -> dict:
"""Dispatch ResearchAgent for each vendor."""
results = {}
for vendor in vendors:
results[vendor] = self.research_agent.run(vendor, criteria)
return results

def evaluate(self, request: dict) -> dict:
vendors = request["vendors"]
criteria = request.get("criteria", {})
weights = request.get("weights", {})

# Step 1: Research
research_data = self._dispatch_research(vendors, criteria)
# Step 2: Scoring
scored_vendors = self._dispatch_scoring(research_data, weights)
# Step 3: Risk Assessment
risk_data = self._dispatch_risk(scored_vendors)
# Step 4: Report Generation
report_path = self._dispatch_report(
{"research": research_data, "scores": scored_vendors, "risks": risk_data},
{"requester": request.get("requester")}
)
return {
"status": "completed",
"recommended_vendor": scored_vendors.get("top_vendor"),
"report_path": report_path,
"scores": scored_vendors,
"risks": risk_data,
}

Research Agent (agents/research_agent.py)

ResearchAgent uses three @tool functions to gather vendor data. In production, these connect to G2 API, Crunchbase, or web scraping pipelines.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
RESEARCH_SYSTEM_PROMPT = """You are a Vendor Research Specialist AI agent.
For each vendor, research and return:
- Company overview (founded, HQ, employee count, funding stage)
- Product features and capabilities relevant to the evaluation criteria
- Pricing model (subscription, usage-based, enterprise licensing)
- Customer reviews and satisfaction scores (G2, Gartner Peer Insights)
- Market presence (Fortune 500 customers, industry certifications)
- Recent news (acquisitions, product launches, layoffs, security incidents)"""


@tool
def search_vendor_reviews(vendor_name: str, category: str) -> dict:
"""Fetches vendor reviews from G2/Gartner. Replace with real API calls in production."""
return {
"overall_rating": 4.2,
"total_reviews": 847,
"ease_of_use": 4.1,
"customer_support": 3.9,
"value_for_money": 3.8,
"feature_rating": 4.3,
"recommended_pct": 82,
"top_pros": ["Easy integration", "Strong API", "Good documentation"],
"top_cons": ["Steep learning curve", "Pricing not transparent"],
}

@tool
def fetch_vendor_pricing(vendor_name: str) -> dict:
"""Fetches vendor pricing. Replace with vendor API or pricing intelligence tool."""
return {
"model": "Per-seat SaaS subscription",
"starter_tier": "$49/user/month (up to 10 users)",
"enterprise_tier": "Custom pricing (100+ users)",
"annual_discount": "20% discount for annual commitment",
"implementation_fee": "$5,000 - $25,000 (one-time)",
}

@tool
def check_certifications(vendor_name: str) -> dict:
"""Checks vendor compliance certifications."""
return {
"hipaa_compliant": True,
"soc2_type2": True,
"iso_27001": True,
"gdpr_compliant": True,
"fedramp": False,
"baa_available": True,
"last_audit_date": "2024-Q3",
"data_residency_options": ["US-East", "US-West", "EU-Frankfurt"],
}

class ResearchAgent:
def __init__(self):
self.model = BedrockModel(
model_id="us.amazon.nova-pro-v1:0",
region_name="us-east-1",
temperature=0.1,
)
self.agent = Agent(
model=self.model,
system_prompt=RESEARCH_SYSTEM_PROMPT,
tools=[search_vendor_reviews, fetch_vendor_pricing, check_certifications],
)

def run(self, vendor_name: str, criteria: dict) -> dict:
prompt = f"""
Research the vendor "{vendor_name}" for enterprise procurement evaluation.
Focus areas: {json.dumps(criteria, indent=2)}
Use all available tools, then return a structured JSON vendor profile.
"""

response = self.agent(prompt)
return {"vendor_name": vendor_name, "raw_response": str(response)}

Scoring Agent (agents/scoring_agent.py)

ScoringAgent applies the weighted MCDA model. compute_weighted_score normalizes scores 0–10 and returns a composite with a letter grade; rank_vendors sorts by composite; identify_score_gaps reveals where vendors diverge most.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
@tool
def compute_weighted_score(vendor_scores: dict, weights: dict) -> dict:
"""Computes weighted composite score for a vendor."""
total_weight = sum(weights.values())
normalized_weights = {k: v / total_weight for k, v in weights.items()}

weighted_scores = {}
composite = 0.0

for criterion, raw_score in vendor_scores.items():
weight = normalized_weights.get(criterion, 0)
weighted = raw_score * weight
weighted_scores[criterion] = {
"raw_score": raw_score,
"weight": weight,
"weighted_score": round(weighted, 3),
}
composite += weighted

return {
"weighted_scores": weighted_scores,
"composite_score": round(composite, 3),
"grade": _score_to_grade(composite),
}

def _score_to_grade(score: float) -> str:
if score >= 8.5: return "A+"
if score >= 8.0: return "A"
if score >= 7.5: return "A-"
if score >= 7.0: return "B+"
if score >= 6.5: return "B"
return "C"

@tool
def rank_vendors(vendor_composites: dict) -> list:
"""Ranks vendors by composite score."""
ranked = sorted(vendor_composites.items(), key=lambda x: x[1], reverse=True)
return [{"rank": i+1, "vendor": v, "score": s} for i, (v, s) in enumerate(ranked)]

@tool
def identify_score_gaps(vendor_scores_matrix: dict) -> dict:
"""Identifies largest score gaps between vendors per criterion."""
criteria = set()
for scores in vendor_scores_matrix.values():
criteria.update(scores.keys())

gaps = {}
for criterion in criteria:
scores_for_criterion = {v: s.get(criterion, 0) for v, s in vendor_scores_matrix.items()}
max_vendor = max(scores_for_criterion, key=scores_for_criterion.get)
min_vendor = min(scores_for_criterion, key=scores_for_criterion.get)
gaps[criterion] = {
"best_vendor": max_vendor,
"best_score": scores_for_criterion[max_vendor],
"worst_vendor": min_vendor,
"worst_score": scores_for_criterion[min_vendor],
"gap": round(scores_for_criterion[max_vendor] - scores_for_criterion[min_vendor], 2),
}
return gaps

Risk Agent (agents/risk_agent.py)

RiskAgent assesses three risk dimensions and computes a composite risk score with a severity label (LOW / MEDIUM / HIGH / CRITICAL).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
@tool
def assess_compliance_risk(vendor_name: str, certifications: dict, industry: str = "healthcare") -> dict:
"""Assesses compliance risk based on certifications and industry requirements."""
risk_score = 0
flags = []

industry_requirements = {
"healthcare": ["hipaa_compliant", "soc2_type2", "baa_available"],
"finance": ["soc2_type2", "pci_dss", "iso_27001"],
"general": ["soc2_type2", "gdpr_compliant"],
}

required = industry_requirements.get(industry, industry_requirements["general"])
for req in required:
if not certifications.get(req, False):
flags.append(f"Missing required certification: {req}")
risk_score += 3

return {
"compliance_risk_score": min(10, risk_score),
"compliance_flags": flags,
"audit_currency": certifications.get("last_audit_date", "unknown"),
}

@tool
def assess_financial_risk(vendor_name: str) -> dict:
"""Financial risk: runway, burn rate, acquisition risk. Connect to Crunchbase/D&B in production."""
mock_financial = {
"funding_stage": "Series C",
"burn_rate_runway_months": 24,
"acquisition_risk": "Low — well-capitalized, independent",
}
risk_score = 2
flags = []
if mock_financial["burn_rate_runway_months"] < 12:
risk_score += 4
flags.append("Low runway (<12 months) — financial instability risk")
return {"financial_risk_score": min(10, risk_score), "financial_flags": flags}

@tool
def assess_exit_risk(vendor_name: str, pricing: dict) -> dict:
"""Data lock-in and exit risk assessment."""
risk_score = 0
flags = []
mitigations = []
has_data_export = True
min_contract_months = 12

if not has_data_export:
risk_score += 4
flags.append("No data export capability — high lock-in risk")
else:
mitigations.append("Data export API available")

if min_contract_months >= 24:
risk_score += 3
flags.append(f"Long minimum contract ({min_contract_months} months)")

return {
"exit_risk_score": min(10, risk_score),
"exit_flags": flags,
"exit_mitigations": mitigations,
}

Report Agent (agents/report_agent.py)

ReportAgent generates a full Markdown report with score tables, risk summaries, rankings, and an implementation roadmap.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
@tool
def format_score_table(vendor_scores: dict) -> str:
"""Formats vendor scores as a Markdown comparison table."""
vendors = list(vendor_scores.keys())
all_criteria = sorted(set(
criterion
for scores in vendor_scores.values()
for criterion in scores.get("weighted_scores", {}).keys()
))

header = "| Criterion | " + " | ".join(vendors) + " |"
separator = "|-----------|" + "|".join(["--------"] * len(vendors)) + "|"

rows = []
for criterion in all_criteria:
row = f"| {criterion.replace('_', ' ').title()} |"
for vendor in vendors:
score = vendor_scores[vendor].get("weighted_scores", {}).get(criterion, {}).get("raw_score", "N/A")
row += f" {score} |"
rows.append(row)

composite_row = "| **COMPOSITE** |"
for vendor in vendors:
comp = vendor_scores[vendor].get("composite_score", "N/A")
grade = vendor_scores[vendor].get("grade", "")
composite_row += f" **{comp} ({grade})** |"
rows.append(composite_row)

return "\n".join([header, separator] + rows)

FastAPI Layer (api/main.py)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
from fastapi import FastAPI
from pydantic import BaseModel
from agents.orchestrator import OrchestratorAgent

app = FastAPI(
title="Vendor Evaluation Assistant",
description="Multi-Agent AI System for Enterprise Vendor Procurement Evaluation",
version="1.0.0",
)

orchestrator = OrchestratorAgent()

class VendorEvaluationRequest(BaseModel):
vendors: list[str]
criteria: dict[str, float] = {
"features": 0.3, "security": 0.25, "support": 0.2,
"value": 0.15, "integration": 0.1
}
context: str = "Enterprise software procurement"
requester: str = "Procurement Team"
industry: str = "general"

@app.post("/evaluate")
async def evaluate_vendors(request: VendorEvaluationRequest):
"""
Trigger the full 5-agent vendor evaluation pipeline.
Returns composite scores, risk assessment, rankings, and report path.
"""

result = orchestrator.evaluate(request.model_dump())
return result

@app.get("/agents/status")
async def agent_status():
return {
"pipeline": [
{"agent": "OrchestratorAgent", "role": "Coordinates all agents"},
{"agent": "ResearchAgent", "role": "Vendor intelligence gathering"},
{"agent": "ScoringAgent", "role": "Weighted MCDA scoring"},
{"agent": "RiskAgent", "role": "Compliance & risk assessment"},
{"agent": "ReportAgent", "role": "Report generation"},
],
"model": "us.amazon.nova-pro-v1:0",
"runtime": "AWS AgentCore Runtime",
}

Running the Project

Prerequisites

  • AWS account with Bedrock Nova Pro access enabled in us-east-1
  • Python 3.11+
  • IAM role with bedrock:InvokeModel permissions

Setup

1
2
3
4
5
git clone https://github.com/your-repo/vendor-eval-assistant
cd vendor-eval-assistant
pip install -r requirements.txt
export AWS_DEFAULT_REGION=us-east-1
uvicorn api.main:app --reload --port 8000

Sample API Call

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
curl -X POST http://localhost:8000/evaluate \
-H "Content-Type: application/json" \
-d '{
"vendors": ["Salesforce", "HubSpot", "Microsoft Dynamics"],
"criteria": {
"features": 0.30,
"security": 0.25,
"support": 0.20,
"value": 0.15,
"integration": 0.10
},
"context": "CRM for 500-person enterprise sales team",
"requester": "Procurement Team",
"industry": "general"
}'

Sample Response

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
"status": "completed",
"recommended_vendor": "Salesforce",
"report_path": "outputs/vendor_evaluation_20250625_143022.md",
"scores": {
"rankings": [
{ "rank": 1, "vendor": "Salesforce", "score": 7.84 },
{ "rank": 2, "vendor": "HubSpot", "score": 7.41 },
{ "rank": 3, "vendor": "Microsoft Dynamics", "score": 6.93 }
],
"top_vendor": "Salesforce"
},
"risks": {
"vendor_risks": {
"Salesforce": { "risk_level": "LOW", "composite_risk_score": 2.1 },
"HubSpot": { "risk_level": "LOW", "composite_risk_score": 2.4 },
"Microsoft Dynamics": {
"risk_level": "MEDIUM",
"composite_risk_score": 3.8
}
}
}
}

File Structure

1
2
3
4
5
6
7
8
9
10
11
vendor-eval-assistant/
├── agents/
│ ├── __init__.py
│ ├── orchestrator.py ← Central coordinator + synthesis
│ ├── research_agent.py ← Reviews, pricing, certifications
│ ├── scoring_agent.py ← Weighted MCDA scoring matrix
│ ├── risk_agent.py ← HIPAA/SOC2/financial/exit risk
│ └── report_agent.py ← Markdown report generation
├── api/
│ └── main.py ← FastAPI REST layer
└── requirements.txt

Production Considerations

  • Replace mock @tool functions with real API integrations (G2, Crunchbase, Clearbit)
  • Add async agent dispatch for parallel vendor research across vendors
  • Integrate Amazon Bedrock Knowledge Bases for vendor history and past contracts
  • Add a Human Approval Tool before report finalization for high-stakes decisions
  • Deploy with AgentCore Runtime for managed scaling and observability
  • Store evaluation history in DynamoDB for audit trail and trend analysis
  • Add streaming responses via FastAPI StreamingResponse for long-running evaluations

Why This Pattern Works

The Vendor Evaluation Assistant demonstrates a clean pattern for production agentic AI:
  • Focused Orchestrator — coordinates without doing domain work itself
  • Stateless specialist agents — each agent owns one concern, makes it testable and swappable
  • @tool-decorated functions — explicit, typed, and easy to mock in tests
  • REST API layer — decouples the agent system from any frontend or consumer
Each agent does exactly one thing well. You can swap the ScoringAgent for a more sophisticated model, add a NegotiationAgent, or plug in a different LLM — without touching the rest of the pipeline.
This is the kind of system that saves enterprise procurement teams hundreds of hours per year.

About the Author

Sweta Jha is a Senior Data Scientist and AI/ML Engineer at Carelon Global Solutions (Elevance Health). AWS Community Builder in AI/ML. Google Cloud Agentic AI Hackathon winner 2025. Specializes in GenAI, LLM orchestration, RAG pipelines, Agentic AI, and MLOps across healthcare, BFSI, and enterprise domains.
  • GitHub: https://github.com/its-me-sweta/Vendor_Evaluation_AI_Assistant.git

Built on AWS | Strands Agents SDK | Amazon Bedrock Nova Pro | FastAPI
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article