Why Plain RAG Fails on Complex Questions — and Graph-Based Retrieval Doesn't

A system built on standard retrieval-augmented generation is good at finding text and bad at finding reasons. Ask it something with a single, locatable answer and it performs well. Ask it a question that depends on a chain of cause and effect — why did this metric move, what actually connects these events — and it tends to hand back a pile of documents that each mention the right keywords without ever assembling the actual explanation.

What retrieval is actually doing

Standard RAG works by comparing a question against a database of text chunks and pulling back whichever ones look most similar, then handing those chunks to a model to summarize into an answer. That's a genuinely good mechanism for lookup-style questions, where the answer lives inside one or two passages. It has no real mechanism for connecting information across documents — it can retrieve five things that each contain a piece of the story, but nothing in the architecture ever assembles the pieces into a chain.

That gap is exactly where multi-step "why" questions live. The answer to "why did this happen" is rarely sitting in one paragraph; it's distributed across several documents in a sequence the retrieval step has no way of reconstructing on its own.

The alternative: build the relationships first

Graph-based retrieval takes an extra step before answering anything: it reads the source material once, pulls out the entities (people, products, events, organizations) and the relationships between them, and stores that as an explicit graph rather than as loose text. A question then gets answered by walking that graph — following the actual chain of cause and effect — instead of pattern-matching against raw chunks. The practical difference shows up exactly where plain RAG struggles: the model isn't inferring a relationship might exist somewhere in five documents, it's reading a relationship that's already been made explicit.

This isn't one lab's idea. Independent research groups working on very different problems — production retrieval systems, agent frameworks, and model training approaches to factual knowledge — have converged on some version of the same fix, which is a reasonably strong signal that the underlying idea holds up rather than being a one-off trick.

The finding worth remembering: the graph matters more than the model

One of the more counterintuitive results in this area is that a smaller model paired with a well-built graph can outperform a much larger model working from unstructured text. That flips a common assumption — that a bigger model is the lever to pull when results aren't good enough — on its head for this specific kind of problem. If the retrieval structure is bad, a bigger model mostly gets better at confidently misusing bad information; if the structure is good, even a modest model has the actual relationships in front of it and doesn't have to guess.

Where a model like Claude fits into the pipeline

Building the graph itself used to require a dedicated NLP pipeline — separate tools for entity extraction, relationship extraction, and deduplication. A general-purpose model like Claude collapses a lot of that into a single step: extracting entities and relationships from raw text, translating a plain-English question into a graph query, and explaining the result back in plain English once the query runs. That doesn't make the graph-building step trivial — deduplicating entities and getting the schema right still benefits from a human sanity-check — but it does mean small teams can realistically build one without hiring a specialized research group to do it.

The practical takeaway if you're building anything with real documents

If what you're building only ever answers lookup questions — find me the document that mentions X — plain retrieval is simpler and it's already good enough. The moment your users start asking "why," start noticing patterns across many documents, or need an answer that requires connecting three or four separate facts, that's the signal a knowledge graph is worth the extra setup. Support tickets, internal documentation, and research libraries are the most common places this shows up first, because those are exactly the places where the interesting questions are rarely answered by a single paragraph.

*Read the original thread this piece is based on: Sprytix on X.*