AWS Builder Center

Evaluating Graph-Enhanced Retrieval on HPC: What We Learned Building Verbis Graph

We evaluated Verbis Graph on HPC infrastructure using NFCorpus and SciFact to understand how graph-enhanced retrieval performs at scale. Here are our latest benchmark results, what the metrics tell us, and where we're taking the research next.

We've spent the past months asking a question that is becoming increasingly important as we build production AI systems:
How do we know whether a GraphRAG retrieval layer is actually getting better?
It's easy to demonstrate an impressive answer from an LLM. It's much harder—and more useful—to systematically measure whether the retrieval system underneath it is finding the correct evidence and ranking that evidence early enough to be useful.
At Prodigy AI Solutions, we've been evaluating Verbis Graph, our graph-enhanced retrieval engine, on HPC infrastructure using standardized information-retrieval benchmarks.
We have now published the complete evaluation report.

Why we focused on retrieval

For a RAG or GraphRAG system, retrieving a relevant document isn't enough.
We want to understand:
Did we retrieve relevant evidence?
How much of the available relevant evidence did we retrieve?
How early did the correct evidence appear?
That led us to focus on three primary metrics: Hit Rate, Mean Recall, and Mean Reciprocal Rank (MRR).
Our latest NFCorpus evaluation produced:
✅ Hit Rate: 92.26% ✅ Mean Recall: 46.57% ✅ MRR: 0.578
One result was particularly interesting to us.
In our earlier evaluation, NFCorpus MRR was 0.303. After improving the retrieval pipeline, it increased to 0.578.
That's important because the system isn't simply finding relevant documents more often—it is ranking relevant evidence substantially closer to the top.
For AI applications, this can make a meaningful difference. The evidence presented near the beginning of the retrieval context can directly affect grounding, answer quality, explainability, and ultimately user trust.
Testing on SciFact
We also evaluated Verbis Graph against SciFact, a scientific retrieval benchmark.
The first results were:
✅ Hit Rate: 98.0% ✅ Mean Recall: 98.0% ✅ MRR: 0.601
Looking deeper into ranking performance:
Hit@1: 51.0% Hit@10: 77.7% Hit@100: 92.3% Hit@1000: 98.0%
These results give us another perspective on how graph-enhanced retrieval behaves across different types of datasets.

Why HPC?

Our experiments have been conducted using CINECA's Leonardo HPC infrastructure.
The goal isn't simply to run retrieval on a powerful machine.
We're interested in understanding what happens as graph-based retrieval moves from experiments toward larger datasets and production-scale workloads.
That includes questions around retrieval quality, scalability, ontology-enhanced retrieval, graph construction, and eventually the relationship between compute resources and retrieval performance.

What we're exploring next

The current report is a checkpoint rather than the end of the evaluation.
We're now interested in expanding the work into additional finance, legal, and scientific datasets, as well as measuring the impact of ontology-enhanced retrieval.
Another objective is reproducibility.
GraphRAG is receiving a lot of attention, but comparing different approaches remains difficult when datasets, retrieval configurations, graph construction methods, and evaluation metrics differ between implementations.
We'd like to contribute more measurable results to that discussion.

And where does AWS fit?

In parallel with this research, we're working on making Verbis Graph easier to deploy in enterprise cloud environments.
Verbis Graph is available through  AWS Marketplace .
For us, these are two sides of the same engineering problem:
Can we improve the intelligence of the retrieval layer while also making the surrounding architecture ready for real enterprise workloads?
There's still a lot to learn.
We've published the full methodology, results, and observations so other builders can take a closer look:
I'd particularly like to hear from other AWS builders working with RAG, GraphRAG, knowledge graphs, or retrieval evaluation:
Which benchmarks and metrics have you found most useful for predicting how a retrieval system will actually perform in production?
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article