Evaluating Verbis Graph on Leonardo HPC: Retrieval Performance Improvements
Every benchmark helps us better understand how graph-based retrieval behaves at scale, and how to build more accurate, explainable, and enterprise-ready AI systems.
Over the past several weeks, we have been evaluating Verbis Graph, our graph-based retrieval engine, on the Leonardo High Performance Computing (HPC) infrastructure at CINECA. Our objective is to better understand how GraphRAG architectures behave at scale while systematically improving retrieval quality before introducing an ontology layer.
Initial Baseline
Our first evaluation was performed on the NFCorpus biomedical information retrieval benchmark using the baseline GraphRAG retrieval pipeline.
Environment
- Infrastructure: Leonardo Booster (CINECA)
- GPUs: NVIDIA A100
- Dataset: NFCorpus (BEIR)
The graph construction process completed in approximately 16 hours, producing a knowledge graph with:
- 22,669 entities
- 34,706 relationships
- Largest connected component: ~19,235 nodes
Retrieval performance:
- Hit Rate: 89.16%
- Mean Recall: 44.56%
- MRR: 0.303
This established our initial retrieval baseline before further optimization.
Retrieval Pipeline Improvements
Following the baseline evaluation, we focused on improving the retrieval layer without modifying the underlying graph construction process.
The main improvements included:
- retrieval pipeline optimization
- lexical retrieval layer integration
- ranking improvements
- GraphRAG architecture maintained
- ontology layer intentionally disabled
Running the same benchmark again produced measurable improvements across all major retrieval metrics.
| Metric | Baseline | Improved |
|---|---|---|
| Hit Rate | 89.16% | 92.26% |
| Mean Recall | 44.56% | 46.57% |
| MRR | 0.303 | 0.578 |
The increase in MRR was particularly encouraging, improving by approximately 91%. Since MRR measures how early the first relevant document appears in the ranked results, this indicates that relevant documents are now consistently retrieved much closer to the top of the ranking.
Additional retrieval statistics include:
- Hit@1: 43.3%
- Hit@10: 72.4%
- Hit@100: 83.9%
- Hit@1000: 92.3%
Scientific Validation
To verify that the improvements generalized beyond a single benchmark, we evaluated the same retrieval pipeline on the SciFact scientific evidence retrieval benchmark.
Results:
- Hit Rate: 98.0%
- Mean Recall: 98.0%
- MRR: 0.601
Additional retrieval statistic include:
• Hit@1: 51.0%
• Hit@10: 77.7%
• Hit@100: 92.3%
• Hit@1000: 98.0%
More than half of all benchmark queries retrieved a relevant scientific document as the first result (Hit@1 = 51%).
What Comes Next
These experiments were intentionally conducted without an ontology layer. The objective was to quantify the impact of retrieval pipeline improvements independently before introducing semantic reasoning.
Our next research phase will focus on:
- ontology-enhanced retrieval
- graph traversal optimization
- multi-hop reasoning
- evaluation on financial and legal retrieval benchmarks
- larger-scale experiments on HPC infrastructure
One of our longer-term goals is to explore how Graph-based retrieval systems scale on production-grade HPC environments using standardized retrieval benchmarks. While GraphRAG research has advanced rapidly, there are still relatively few published evaluations that combine large-scale HPC infrastructure with reproducible retrieval benchmarking. We hope these experiments contribute useful insights to both the GraphRAG and HPC communities.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article