Beyond Vector Search: Why We Built with GraphRAG
The Promise and the Problem
Vector databases changed everything about how we build AI applications. The ability to encode meaning into mathematical space and find semantically similar content felt like magic when we first implemented it. Type a question in plain English, and the system retrieves relevant documents based on meaning rather than keywords. For many use cases, this approach works brilliantly.
We built several client projects on this foundation. Document search, FAQ matching, content recommendation systems. The results impressed stakeholders and delivered genuine value. Vector search became our default approach for any retrieval problem.
Then we hit a wall.
A client asked us to build an internal knowledge system for their organisation. The requirement seemed straightforward: help employees find information about projects, people, and processes. We deployed a vector database, embedded their documentation, and demonstrated the initial results. Everyone was pleased. Then someone asked a simple question that exposed a fundamental limitation.
“How is Sarah connected to the Birmingham expansion project?”
The system returned nothing useful. It found documents mentioning Sarah. It found documents about the Birmingham project. But it could not answer the actual question because the connection existed across multiple documents and required reasoning about relationships, not similarity.
Understanding Why Vector Search Falls Short
Vector search operates on a simple principle: convert text into numerical representations called embeddings, then find other embeddings that sit nearby in mathematical space. When you search for “project management tools,” the system finds documents about Trello, Asana, and similar topics because their embeddings cluster together.
This works wonderfully for questions with direct answers. “What is our refund policy?” returns the refund policy document. “How do I reset my password?” surfaces the relevant help article. The semantic similarity between question and answer ensures accurate retrieval.
The approach fails for questions requiring multiple reasoning steps. Consider what happens when you ask about connections between entities. The system must first identify that Sarah is a senior architect, then find that she authored the technical specification for the Birmingham project, then discover that specification influenced the procurement decisions. No single document contains this complete chain. Vector search finds isolated pieces but cannot assemble them into understanding.
We call these “multi-hop” reasoning questions because answering them requires hopping between multiple pieces of information. Traditional retrieval treats each document as an island. Real organisational knowledge exists as a connected web where meaning emerges from relationships as much as content.
GraphRAG: Connecting the Dots
GraphRAG stands for Graph-based Retrieval Augmented Generation. The name sounds technical, but the concept mirrors how humans actually think about connected information. Rather than treating documents as isolated items, GraphRAG builds a map of entities and their relationships.
Think of it as the difference between a library catalogue and a detective’s investigation board. The catalogue tells you which shelf holds each book. The investigation board shows you how every person, place, and event connects to every other. When someone asks about Sarah and the Birmingham project, GraphRAG traverses the relationship map rather than searching for similar text.
The implementation adds a knowledge graph layer alongside your vector database. As documents enter the system, natural language processing extracts entities (people, projects, locations, concepts) and relationships (authored, approved, mentioned, collaborated). These become nodes and edges in a graph database. The vector database still handles semantic similarity. The graph database handles structural relationships.
When a question arrives, the system determines whether it requires simple retrieval or relationship traversal. Simple questions go straight to vector search. Complex questions trigger graph queries that walk relationship paths before gathering relevant context for the language model.
The language model receives richer context because it sees not just similar documents but the actual relationship chain connecting query entities. “Sarah is connected to Birmingham through her authorship of Technical Specification 47, which established requirements that influenced Procurement Decision 892, which approved the Birmingham expansion scope.” The model can now provide a coherent answer that no single document contains.
The Technical Stack We Use
Pinecone
Pinecone serves as our vector database layer. We chose Pinecone for its managed infrastructure, excellent query performance, and straightforward scaling. Documents are chunked, embedded using models like OpenAI's text-embedding-3-large, and indexed for semantic search. When questions require similarity matching rather than relationship traversal, Pinecone handles retrieval directly.
Neo4j
Neo4j provides our graph database layer. Neo4j's Cypher query language makes relationship traversal intuitive, and its visualisation tools help during development and debugging. As documents enter the system, our extraction pipeline identifies entities and relationships, creating nodes and edges that mirror the organisational knowledge structure. Graph queries can find paths between entities, identify clusters, and surface indirect connections.
Microsoft's GraphRAG Framework
Microsoft's GraphRAG Framework offers an alternative approach we deploy for certain use cases. Rather than building custom extraction pipelines, Microsoft's framework automatically constructs knowledge graphs from document collections and provides pre-built query strategies. This accelerates deployment when project timelines are tight, though custom Neo4j implementations offer more control for complex requirements.
Measuring the Difference
GraphRAG achieved 78% accuracy on the same questions. The relationship traversal provided structural context that enabled coherent multi-hop answers. Even when answers were incomplete, they typically provided useful partial paths rather than disconnected fragments.
When to Choose GraphRAG
GraphRAG adds complexity and cost. The additional infrastructure, extraction pipelines, and maintenance overhead mean it should not be the default choice for every retrieval application.
Choose vector search alone when your questions have direct answers contained within single documents, when your content lacks strong entity relationships, or when latency requirements preclude graph traversal. FAQ systems, documentation search, and content recommendation often work excellently with vectors alone.
Choose GraphRAG when users ask questions about connections between entities, when answers require synthesising information across multiple sources, when your domain has rich relationship structures (organisational hierarchies, project dependencies, regulatory frameworks), or when reasoning quality matters more than response speed.
Many production systems benefit from a hybrid approach that routes queries to the appropriate retrieval strategy based on question analysis. This provides vector search speed for simple questions while enabling graph reasoning when complexity demands it.
We have found GraphRAG particularly valuable for professional services firms tracking client relationships, manufacturing companies managing supplier networks, and any organisation where understanding "how things connect" delivers business value beyond "what documents exist."
Moving Forward
Building effective GraphRAG systems requires careful attention to entity extraction quality, relationship schema design, and query routing logic. The technology decisions matter less than the knowledge modelling work that precedes them. If your organisation struggles with questions that span multiple documents, or if your users express frustration that the AI “knows” information but cannot connect it, GraphRAG may provide the architectural shift you need. We are happy to discuss whether this approach fits your specific requirements.
Ready to Activate Your Digital Advantage?
Interested in finding out more? Chat to Our Intelligent Assistant Now to Discover What We Can Do for You.
Discover more AI Insights and Blogs
By 2027, your biggest buyer might be an AI. How to restructure your Ecommerce APIs and product data so "Buyer Agents" can negotiate and purchase from your store automatically
Dashboards only show you what happened. We build Agentic Supply Chains that autonomously reorder stock based on predictive local trends, weather patterns, and social sentiment
Stop building static pages. Learn how we configure WordPress as a "Headless" receiver for AI agents that dynamically rewrite content and restructure layouts for every unique visitor
One agent writes, one edits, one SEO-optimizes, and one publishes. How we build autonomous content teams inside WordPress that scale your marketing without scaling your headcount
One model doesn't fit all. We break down our strategy for routing tasks between heavy reasoners (like GPT-4) and fast, local SLMs to cut business IT costs by 60%
Don't rewrite your old code. How we use Multi-Modal agents to "watch" and operate your legacy desktop apps, creating modern automations without touching the source code
You wouldn't give an intern root access to your database. Why are you giving it to ChatGPT? Our framework for "Role-Based Access Control" in Agentic Systems