Building a Repeatable Vector Search Benchmark on AWS

Compare Amazon S3 Vectors, Amazon OpenSearch Serverless, Amazon Aurora PostgreSQL with pgvector, and Amazon Neptune Analytics using the same embeddings, queries, and measurement methodology.

Yevgeni Shapiro

Head Of Cloud | Driving Cloud-Native Innovation | Platform Engineering

Published Oct 1, 2026

Last modified Oct 1, 2026

Vector search is now a common building block for retrieval-augmented generation (RAG), semantic search, recommendations, and AI-assisted applications. On AWS, developers can choose from several architectures for storing and searching embeddings.

The challenge is not finding a service that can perform vector search. The harder question is:

How does a particular vector-storage architecture behave for my workload, and what does that behavior cost?

This article presents a repeatable benchmark architecture that addresses that question. It creates one common embedding corpus, loads the same vectors into four AWS vector-search architectures, and then runs the same query workload automatically on a schedule. The benchmark records p50, p95, and p99 latency in Amazon CloudWatch so that performance can be analyzed alongside workload-specific cost modeling.

The architecture compares:

Amazon S3 Vectors
Amazon OpenSearch Serverless
Amazon Aurora PostgreSQL with pgvector
Amazon Neptune Analytics

The goal is not to declare one service universally better than another. The goal is to create a consistent measurement framework that lets teams evaluate the architecture that fits their own workload.

1. Create a common embedding corpus

The benchmark begins with a corpus of approximately 50,000 storm narratives.

A developer runs the embedding workflow once. Each narrative is sent to Amazon Bedrock Titan Text Embeddings V2, producing a 1,024-dimensional vector.

Titan Text Embeddings V2 supports output dimensions of 256, 512, or 1,024, with 1,024 as the default. The model accepts text and produces vector representations that can be used for semantic search and related workloads.

The initial workflow is therefore:

1
2
3
4
5
6
7
8
9
10
Developer
    |
    | make_embed
    v
Amazon Bedrock
Titan Text Embeddings V2
    |
    | 1,024-dimensional vectors
    v
~50,000 embedded documents

The embedding step is intentionally performed once.

If every vector database generated its own embeddings, differences in model configuration or embedding generation could contaminate the benchmark. Instead, every store receives the same vector representation.

This establishes the benchmark's first invariant:

Same documents → same embeddings.

2. Load the same vectors into four architectures

The second stage takes the common embedding dataset and loads it into four different AWS services.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
                     Common vectors
                           |
          +----------------+----------------+
          |                |                |
          v                v                v
     S3 Vectors       OpenSearch         Aurora
                      Serverless         PostgreSQL
                                            |
                                            v
                                         pgvector

                           |
                           v
                    Neptune Analytics

The ingestion process should use equivalent metadata, vector dimensions, and similarity semantics wherever the respective services support them.

Amazon S3 Vectors

The first implementation uses Amazon S3 Vectors.

S3 Vectors introduces vector buckets and vector indexes specifically for storing and querying embeddings. A vector index defines properties such as vector dimension and distance metric, and vectors can have associated metadata for filtering.

A similarity query uses the QueryVectors API and specifies a query vector, the number of results to return, and the target vector index. Metadata filters can also be applied.

For the benchmark, the logical flow is:

1
2
3
4
5
6
7
8
9
10
Embedding
    |
    v
S3 vector bucket
    |
    v
Vector index
    |
    v
QueryVectors

S3 Vectors is particularly interesting in this comparison because it provides purpose-built vector storage without requiring the benchmark to provision a conventional database or search cluster. AWS describes it as cost-optimized vector storage for applications such as RAG, semantic search, and AI agents.

Amazon OpenSearch Serverless

The second implementation uses Amazon OpenSearch Serverless.

OpenSearch Serverless provides a managed environment for search workloads and supports vector search. AWS also provides vector ingestion capabilities that can create and manage ingestion pipelines for OpenSearch domains and OpenSearch Serverless collections.

The benchmark stores the same vectors in a vector-search collection:

1
2
3
4
5
6
7
8
9
10
Embedding
    |
    v
OpenSearch Serverless
    |
    v
Vector search collection
    |
    v
k-NN query

This provides a useful comparison against S3 Vectors because OpenSearch brings a broader search engine model to the problem, including capabilities that can be useful when vector search needs to coexist with other search requirements.

Amazon Aurora PostgreSQL with pgvector

The third implementation uses Amazon Aurora PostgreSQL-Compatible Edition with the open-source pgvector extension.

This approach is especially relevant when an application already uses PostgreSQL and needs vector search alongside relational data.

Conceptually, the schema might contain:

1
2
3
4
5
6
CREATE TABLE storm_documents (
    id          BIGSERIAL PRIMARY KEY,
    narrative   TEXT,
    embedding   vector(1024),
    event_date  DATE
);

The vector is stored as part of the relational record rather than in a separate vector-search system.

AWS documents Aurora PostgreSQL support for pgvector, including exact and approximate nearest-neighbor search, cosine distance, L2 distance, inner product, and HNSW and IVFFlat indexing options.

The resulting architecture looks like:

1
2
3
4
5
6
7
8
Embedding
    |
    v
Aurora PostgreSQL
    |
    +--> relational predicates
    |
    +--> pgvector similarity search

That combination is an important part of the benchmark because vector retrieval does not always exist in isolation. An application might need to retrieve semantically similar documents while simultaneously applying relational conditions.

3. Amazon Neptune Analytics

The fourth implementation uses Amazon Neptune Analytics.

Neptune Analytics adds vector similarity search to a graph-oriented data model. Embeddings can be associated with graph nodes, allowing vector similarity to be combined with graph relationships and other graph context.

This creates a different retrieval model:

1
2
3
4
5
6
7
8
9
10
11
12
13
                 Query vector
                      |
                      v
             Vector similarity
                      |
                      v
              Relevant nodes
                      |
                      v
               Graph traversal
                      |
                      v
              Additional context

This can be useful when semantic similarity is only one part of the retrieval problem.

For example, a support application might first find documents semantically similar to a question and then traverse graph relationships to identify related products, previous cases, authors, or runbooks. AWS specifically describes this combination of vector similarity and graph context as a use case for Neptune Analytics.

Neptune Analytics provides vector-similarity search algorithms for approximate nearest-neighbor retrieval, including HNSW-based vector search.

4. Schedule the benchmark

After ingestion, the benchmark becomes automated.

Amazon EventBridge Scheduler invokes the benchmark Lambda every six hours:

1
2
3
4
5
EventBridge Scheduler
       |
       | rate(6 hours)
       v
Benchmark Lambda

EventBridge Scheduler supports recurring rate and cron expressions and can invoke Lambda asynchronously. AWS recommends EventBridge Scheduler for scheduled invocations rather than legacy scheduled EventBridge rules.

The six-hour interval is an example rather than a requirement. A production implementation could choose a different interval based on how frequently the team wants to sample the systems.

The important property is that the same benchmark executes repeatedly without manual intervention.

5. Lambda is the common benchmark client

The benchmark Lambda provides a single control point for the experiment.

The reference architecture uses:

ARM64
Python 3.13
AWS Lambda Powertools
AWS SDK clients
Amazon CloudWatch for metrics and logs

During each run, Lambda obtains the embedding for the canonical query using the same embedding model used during corpus generation.

It then sends that query vector to every backend.

Conceptually:

1
2
3
4
5
6
7
8
9
query_vector = embed_query(canonical_query)

s3_result = query_s3(query_vector)

opensearch_result = query_opensearch(query_vector)

aurora_result = query_aurora(query_vector)

neptune_result = query_neptune(query_vector)

Each backend is implemented behind a common logical interface.

For example:

1
2
3
4
5
VectorStore
    |
    +-- load()
    |
    +-- query()

The implementations become:

1
2
3
4
S3VectorsStore
OpenSearchStore
AuroraPgVectorStore
NeptuneStore

This makes the benchmark extensible.

If another vector service needs to be evaluated later, the benchmark can add another adapter rather than redesigning the entire harness.

6. Keep the queries equivalent

A benchmark is only useful if the queries are comparable.

For every backend, try to hold the following constant:

Parameter	Benchmark requirement
Corpus	Same documents
Embeddings	Same model/output
Dimensions	Same dimension
Query text	Same canonical queries
Query embeddings	Same query vector
Top K	Same result count
Distance metric	Equivalent configuration
Filters	Same logical filters
Measurement	Same timing boundary

For example, if the benchmark requests the 10 nearest neighbors, the equivalent query should request 10 results from each system.

This is more important than making the service APIs look identical. Different systems expose different query languages and indexing implementations, but the logical workload should be equivalent.

7. Measure p50, p95, and p99

The benchmark should not report only average latency.

Instead, record at least:

p50 — median latency
p95 — 95th percentile
p99 — 99th percentile

These measurements answer different questions.

p50

What does a typical query experience look like?

p95

How does latency behave for slower requests?

p99

What happens in the long tail?

For an interactive AI application, tail latency can matter significantly. A system with a good median but occasional very slow searches can produce a different application experience from one with a more predictable latency distribution.

8. Publish metrics using CloudWatch EMF

The benchmark Lambda records the latency measurements using CloudWatch Embedded Metric Format (EMF).

EMF allows application metrics to be written alongside structured log events. CloudWatch extracts the metric values automatically, while the original logs remain available for deeper analysis with CloudWatch Logs Insights.

A logical metric model could look like:

1
2
3
4
5
6
7
8
9
Namespace: VectorBenchmark

Dimensions:
    Store

Metrics:
    p50LatencyMs
    p95LatencyMs
    p99LatencyMs

The resulting data might look conceptually like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Store = S3Vectors
    p50 = ...
    p95 = ...
    p99 = ...

Store = OpenSearch
    p50 = ...
    p95 = ...
    p99 = ...

Store = Aurora
    p50 = ...
    p95 = ...
    p99 = ...

Store = Neptune
    p50 = ...
    p95 = ...
    p99 = ...

One important operational consideration is metric cardinality. CloudWatch notes that every unique dimension combination can create a separate custom metric, so dimensions such as request IDs should generally not be used as metric dimensions.

For this benchmark, Store is a useful low-cardinality dimension.

9. Separate performance measurements from cost modeling

The architecture deliberately doesn't hard-code a universal "winner."

Instead, it produces measurements that can be combined with the assumptions of the workload.

A benchmark report might eventually contain:

Architecture	p50	p95	p99	Monthly modeled cost
S3 Vectors	measured	measured	measured	modeled
OpenSearch Serverless	measured	measured	measured	modeled
Aurora + pgvector	measured	measured	measured	modeled
Neptune Analytics	measured	measured	measured	modeled

The cost column should be calculated from the actual workload assumptions and current AWS pricing, rather than inferred from latency.

# vector-database # aws-lambda # opensearch # amazon-bedrock # amazon-rds

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Building a Repeatable Vector Search Benchmark on AWS

Yevgeni Shapiro

1. Create a common embedding corpus

2. Load the same vectors into four architectures

Amazon S3 Vectors

Amazon OpenSearch Serverless

Amazon Aurora PostgreSQL with pgvector

3. Amazon Neptune Analytics

4. Schedule the benchmark

5. Lambda is the common benchmark client

6. Keep the queries equivalent

7. Measure p50, p95, and p99

8. Publish metrics using CloudWatch EMF

9. Separate performance measurements from cost modeling