Beyond Vector Search: Building and Benchmarking Enterprise GraphRAG Architectures

Retrieval-Augmented Generation (RAG) has become the standard pattern for grounding Large Language Models (LLMs) in enterprise domain knowledge. However, standard RAG architectures—relying primarily on text chunking and vector similarity search—exhibit severe structural limitations when answering complex, multi-hop queries across highly interlinked data.

When a query requires traversing explicit dependencies—such as identifying which top-tier customer orders will be impacted if a primary parts supplier delays a shipment—vector embeddings fail. Vector search retrieves semantically similar text fragments, but it cannot navigate structural correlations, parent-child dependencies, or multi-hop relational paths. Standard vector similarity fails when answering questions that require traversing explicit, multi-hop dependencies across disconnected systems:

• Pharmaceutical Care: Determining medication safety requires synthesizing patient histories, prescription lists, drug interaction databases, and contraindication guidelines. GraphRAG traverses the explicit relational path between patient conditions and chemical interactions.
• Logistics & Supply Chain: Evaluating how a delayed parts supplier impacts top-tier customer orders requires linking supplier schedules, port capacity reports, shipping manifests, and customer SLAs across supply chain tiers.
• Aviation Engineering: Clearing an aircraft for flight based on engine wear requires cross-referencing maintenance logs, OEM technical manuals, flight plans, and engineering team rosters.
• Financial Services & AML: Validating high-value wire transfers requires correlating anti-money laundering policies, real-time fraud detection feeds, customer profiling histories, and device fingerprinting logs.

By capturing explicit entities and relationships within a Knowledge Graph, GraphRAG replaces probabilistic semantic matching with deterministic structural traversal.

GraphRAG bridges this structural gap by combining the relational rigor of Knowledge Graphs (KGs) with the linguistic capabilities of LLMs. Grounded in Euranova's peer-reviewed research—"Quantifying Retrieval Quality in GraphRAG: A Schema-Agnostic Approach" and "Evaluation of GraphRAG Strategies for Efficient Information Retrieval" (authored by Thibaud Vanmechelen, Alexandre Achten, Zaineb Gabsi, and Sabri Skhiri)—this technical deep dive outlines the architecture, construction strategies, and empirical retrieval benchmarks for production-ready GraphRAG systems.

The Relational Limit of Standard RAG

Standard RAG operates on flat vector spaces. Documents are split into chunks, converted into dense vector embeddings, and retrieved based on cosine similarity to the user's prompt.

While effective for isolated fact retrieval ("What is our return policy?"), standard RAG fails under three common enterprise conditions:

1. Multi-Hop Traversal: The answer depends on a chain of relationships ($A \rightarrow B \rightarrow C \rightarrow D$) where the intermediate nodes do not share high semantic similarity with the initial query.
2. Topological Aggregation: The query requires calculating or summarizing structural properties across subgraphs (e.g., counting total downstream dependencies).
3. Global Relationship Reasoning: Synthesizing connectivity patterns across disjointed document collections.

[Standard Vector RAG]
User Query ---> Embedding ---> Vector Search (Top-K Chunks) ---> LLM ---> Answer
(Fails when context is spread across interlinked relational paths)

[GraphRAG]
User Query ---> Entity/Relation Entry ---> Graph Traversal / Subgraph Extraction ---> Graph Context + LLM ---> Precise Answer

By explicitly representing entities as nodes and relationships as edges, GraphRAG converts implicit document connections into a deterministic graph structure that can be queried and traversed.

Knowledge Graph Construction Architecture

Transforming unstructured enterprise documents (e.g., PDFs, technical reports, operational manuals) into a structured Knowledge Graph requires a strict, multi-stage ingestion pipeline.

Unstructured PDFs 
   │
   ▼
[Stage 1: Preprocessing] ──► Docling (PDF to Markdown) ──► LangChain Chunking
   │
   ▼
[Stage 2: Entity Extraction] ──► LLM Entity Extraction ──► Type Clustering & Deduplication
   │
   ▼
[Stage 3: Relation Extraction] ──► LLM Relationship Extraction ──► Semantic Clustering
   │
   ▼
[Stage 4: Post-Processing] ──► Aggregation & KG Construction ──► Graph Database

Ingestion Pipeline Stages

1. Document Preprocessing: Raw PDF documents are parsed into structured Markdown using Docling, preserving layout context before being segmented into text chunks via LangChain.
2. Entity Extraction & Typing: An LLM identifies named entities within chunks, performing semantic type clustering and deduplicating extracted entities against a cached type ontology.
3. Relation Extraction: Relationships between extracted entities are mapped using LLM-guided relation identification, followed by semantic clustering to standardize edge predicates.
4. Post-Processing & Graph Consolidation: Extracted nodes and edges are aggregated, resolved, and committed to the graph database.

Construction Trade-Offs: Schema-Constrained vs. Schema-Free

We evaluated two distinct approaches to graph construction:

• Schema-Constrained Construction: Enforces a predefined domain ontology. While this reduces noise and extraction costs, it requires upfront domain modeling.
• Schema-Free Construction: Allows the LLM to extract entities and relations dynamically. This yields high adaptability but increases downstream node duplication and ontology noise.

In addition, the structural placement of source text chunks within the graph topology dictates overall retrieval performance. Using our internal GraphRAGBench benchmark, we evaluated two topological strategies across schema-constrained and schema-free pipelines:

• "Text Chunks on the Side": Chunks exist outside the primary graph hierarchy, linked as metadata.
• "Text Chunks as Nodes": Chunks are first-class structural nodes connected directly to extracted entities.

Key Finding: Modeling "Text Chunks as Nodes" consistently yields superior relevance, utilization, and completeness metrics across all schema configurations. Preserving text chunks as explicit structural entities provides direct anchor points during subgraph retrieval.

Cost Analysis (Graph Construction)

Using gpt-5.4-mini (at an average density of ~1,000 tokens per document page), processing costs scale predictably:

• Schema-Free Extraction: $16.00 – $17.00 per 1M tokens ($0.016 / page)
• Schema-Constrained Extraction: $8.00 per 1M tokens ($0.008 / page)

Predefining an ontology cuts LLM processing costs by roughly 50% by restricting the token footprint required for open-ended entity and relationship classification.

Evaluating Retrieval Quality: The Schema-Agnostic Benchmark

A critical flaw in standard RAG benchmarks (e.g., RAGAS, ARES) is diagnostic conflation. By evaluating only the final LLM-generated answer, existing frameworks cannot isolate whether a failure occurred during context retrieval or response generation.

Furthermore, current benchmarks suffer from:

1. Topological Blindness: Measuring semantic similarity rather than verifying whether the retriever traversed the correct graph paths.
2. Ground Truth Incompleteness: Rewarding partial keyword matches without verifying if the full subgraph context was retrieved.
3. Lack of Structural Ground Truth: Absence of exact node/edge target sets for the retrieval step.

Euranova's Evaluation Methodology

To quantify internal retrieval accuracy independently of LLM synthesis, we developed a schema-agnostic evaluation framework using synthetic dataset generation grounded in deterministic Cypher queries across a 9-category taxonomy (Out-of-Scope, Direct Node, Negative Node, Multi-Hop, Intersection-Node, Direct-Edge, Negative-Edge, Aggregation Node, and Aggregation Edge). Ground truth element sets (nodes and edges) are extracted directly via Cypher, enabling exact calculation of Precision, Recall, F1-Score, and topological Proxy metrics without relying on LLM-as-a-judge approximations.

[1. Taxonomy & Templates] ──► Establish 9 query categories using generalized Cypher templates
            │
            ▼
[2. Query Population]     ──► Populate templates with KG data; execute Cypher for exact ground truth
            │
            ▼
[3. Question Generation]  ──► LLM translates Cypher to natural language (varied writing styles)
            │
            ▼
[4. Evaluation Execution] ──► Retriever output IDs compared directly against Cypher ground truth

Dataset Baseline

The benchmark was executed against a structured subset of the Hetionet biomedical database:

• Total Nodes: 38,584 (82% of full Hetionet)
• Total Edges: 1,488,879 (66% of full Hetionet)

Evaluated Retriever Architectures

We benchmarked four distinct retrieval strategies:

1. Random Baseline: Selects $k$ random nodes/edges from the graph schema to establish a lower-bound performance threshold.
2. Graph Traversal (Text2Cypher): An LLM translates natural language queries directly into executable, schema-aware Cypher queries.
3. Semantic Agent: An LLM-driven agent that uses semantic vector search to identify initial entry nodes, then recursively crawls neighboring edges.
4. Chain-of-Thought (CoT) Agent: An advanced agentic retriever extending the Semantic Agent with a broader toolset, including neighbor counting, common connection discovery, and targeted node/edge lookup.

Empirical Benchmark Results

Global Retriever Performance

Retrieval Recall & Hit Rate Across Architectures

CoT Agent         [=======================================] Recall: 0.89 | Hit Rate: 0.95
Semantic Agent    [==============================>        ] Recall: 0.81 | Hit Rate: 0.81
Graph Traversal   [======================>                ] Higher Precision, Lower Recall
Random Baseline   [===>                                   ] Baseline Floor

• Chain-of-Thought (CoT) Agent: Achieved the highest overall Recall (0.89) and Hit Rate (0.95). Its dynamic tool selection (neighbor counting, common connection discovery, specific entity lookup) enabled multi-step structural reasoning across complex subgraphs.
• Semantic Agent: Reached a Recall of 0.81 and Hit Rate of 0.81. Strong on Out-of-Scope (0.96) and Negative Node (0.75), but underperformed CoT on complex topologies like Intersection-Node (0.15) and Negative-Edge (0.26) due to lacking extended inspection tools.
• Graph Traversal (Text2Cypher): Showed high performance on Aggregation Edge (0.96), Direct Node (0.94), and Aggregation Node (0.87), outperforming Semantic Agent on direct structural lookups. However, it suffered severe timeouts on Negative Edge (0.42) and Direct Edge (0.40) evaluation, lowering its multi-hop recall.
• Random Baseline: Lowest across all categories, with non-zero scores (e.g., Aggregation Node 0.47) driven by structural Proxy metrics that grant partial credit for retrieved candidate counts.

Category-Level Performance Breakdown (CoT Retriever)

Performance varies drastically depending on the topological query pattern:

The Impact of Phrasing and Writing Style

To test robustness, queries were rewritten across three natural language styles: Standard, Paraphrased, and Synonym-heavy.

Result: Variations in writing style had negligible impact on retrieval F1 scores across all agentic retrievers. Structural graph category—not natural language phrasing—is the primary determinant of retrieval performance.

Production Insights & Engineering Takeaways

1. Agentic Retrievers Outperform Static Text2Cypher

While Text2Cypher offers deterministic execution, it breaks down when facing ambiguous natural language prompts or deep multi-hop requirements. LLM-driven agentic retrievers utilizing dynamic toolsets (CoT) provide the structural flexibility needed to navigate complex enterprise graph topologies reliably.

2. The "Over-Inclusion" Effect

Strict retrieval metrics often penalize agentic retrievers for "over-inclusion"—retrieving neighbor nodes or structural subgraphs beyond the minimal ground truth target. However, in production pipelines, retrieving adjacent topological context enhances downstream LLM synthesis without degrading final answer accuracy.

Architectural Recommendation: Implement a hybrid evaluation protocol that pairs strict deterministic graph retrieval metrics (Precision/Recall on node IDs) with an LLM-as-a-judge layer to assess context utility during generation.

3. Resource Efficiency vs. Latency

High retrieval accuracy in agentic systems comes at the cost of increased API call iteration and execution latency:

• Text2Cypher: Low latency (single LLM pass + graph execution), but high failure rates on complex schemas.
• CoT Agent: Higher latency and increased API calls per query, but critical for multi-hop reasoning.

Engineers must profile query workload distributions: simple direct lookups should route to Text2Cypher or direct vector entry, reserving full CoT agentic traversal for multi-hop relational queries.