We Tested a Knowledge Graph Properly and Killed It

We gave a knowledge graph tool a fair pilot on 46 real documents, with the kill criteria written down before the first run. It drew mostly true edges. Then we ran it a second time on identical input and only 15.85% of the node labels matched. We turned it off, kept a boring search index, and I think that was the right call.

The problem we were actually trying to solve

Our company brain is a git repo full of markdown. In July 2026 it held about 3,548 markdown files, including 395 meeting transcripts and 545 email threads. Retrieval was grep and reading files. That works when you know the word you’re looking for, and fails when you don’t.

The classic failure: “where have we dealt with pricing pushback?” Nobody writes “pricing pushback” in a transcript. People say “that’s more than we budgeted” or “can we phase this.” On that exact question, grep found zero files containing the phrase.

We also had a link problem. About 22% of the wiki-style links between documents pointed at nothing. Knowledge lived in per-client folders with no way to walk from one to another.

Knowledge graphs are the obvious pitch for this. An LLM reads your documents, pulls out people, companies, projects and relationships, and gives you a graph you can traverse. “Show me every connection between this prospect and someone we already know.” I wanted that to work.

Reading the issue tracker first

Before running anything, I read the tool’s own issue tracker. That turned out to be the most useful part of the whole project. Open issues reported around 30% junk “stub” nodes on prose, entity duplication bloat as high as 62%, overconfident edges on prose, and an incremental update mode that could wipe a graph. Independent measurements of the token savings ranged from negative to 47%, far from the headline claims.

None of that meant the tool was useless. It meant we needed a real test with thresholds, decided in advance, so that I couldn’t talk myself into keeping something I was already excited about. The plan said it plainly: the graph would never be load-bearing truth, and if it failed the audit we’d remove it and revisit in two quarters.

We also added a rule that matters more than it looks. A graph query only counted as useful if our new search index couldn’t have answered it. If both tools answer the same question, the graph is just extra infrastructure.

The pilot

The test corpus was a single partnership folder: 46 documents plus 2 code files, about 2.3 MB. Extraction ran inside our Claude subscription, so the metered cost was $0 (roughly 1.9 million plan tokens). The result was 139 nodes, 202 edges and 15 communities.

The first surprise came before any audit. At a concurrency of 2, two of five extraction chunks failed with empty error messages, and the tool returned a partial graph without saying so. A rerun at concurrency 1 completed. Anything headless would have needed its own failure checks on top.

Then the audit, one check at a time.

Edges traced to a source sentence. We sampled 28 relationships and gave each one to a separate verifier agent whose only job was to find the sentence that supports it. 25 traced cleanly, 1 partially, 2 not at all. That’s 89%, above our 80% bar. The failures were instructive: one edge said a memo “references” a draft agreement. The memo never mentions it. They just lived in the same folder.

Junk nodes. Zero stubs out of 126 semantic nodes. Pass. This was better than the tracker led me to expect.

Duplicate entities. 16.7% of nodes were the same entity minted again from a different document. One person showed up as four separate nodes. Worse, two competing products got merged into one node, which is the kind of error that quietly poisons any answer built on it. The bar was 10%. Fail.

Determinism. We copied the corpus and ran a full fresh extraction. Same documents, same settings. The second graph had 147 nodes instead of 139, and only 15.85% of node labels matched exactly. The entities were mostly the same, but the names drifted: “30/60/90 plan + success scoreboard” in one run, “30/60/90 plan + scoreboard presentation” in the other. A human reads those as the same thing. A join key does not.

Incremental updates. We backed up the graph, edited one file, deleted another, and ran the update command. The deletion was handled. The edit never got re-extracted, because the headless update only reprocesses code. Meanwhile the graph ballooned from 157 to 542 nodes, and the tool printed a message suggesting I run the update from inside an AI assistant instead. Our planned nightly job had a safety check that aborts if node count jumps more than 20%, so it would have caught this. It would also have caught it every single night.

Why quality wasn’t the problem

The extraction quality mostly passed. What failed was everything around it.

Our design was a nightly remote job that updates the graph and commits it to git, so anyone on the team can read it without installing anything. That requires incremental updates (broken) or full re-extraction every night. Full re-extraction is non-deterministic, so the committed graph would rewrite about 85% of itself on every refresh. Nobody can review that diff. No other tool can hold a stable reference to a node that gets renamed every night. And the duplicate problem would have needed a custom cleanup pass, meaning more bespoke code to maintain for a layer whose main selling point overlapped with search we already had.

So the verdict was NO-GO, revisit around January 2027. We uninstalled it, deleted the output folder, and kept the ignore file that defines the curated scope, so a future retry starts where this one stopped. The revisit conditions are written down: upstream fixes for incremental updates, entity merging or stable labels, or three real queries a month that search and our simple link graph demonstrably can’t answer.

What we kept instead

One ruling shaped all of this. Anyone using the brain should need only git, Claude Code, uv and the shared environment file. Heavy jobs run remotely. Nobody installs models on their laptop.

That ruling killed a local hybrid search we’d planned, which needed about 2 GB of models on every machine. It also ruled out Gemini File Search, because it has no plain retrieval endpoint. Every query is a paid generation that returns an answer instead of the source chunks, which is backwards for a system whose whole point is answers with sources.

We landed on Vertex AI Search. It returns chunks and costs about $0 to $6 a month, with a $10 hard cap. We indexed 1,365 documents with zero failures after one fix (it rejects markdown as a content type, so we upload it as plain text). On the “pricing pushback” question where grep found nothing, it returned the actual negotiation transcripts. On another question where grep returned 12 unranked files, it put the right transcripts at the top.

It isn’t magic either. The exact phrasing from our test set scored zero hits, and a more natural phrasing of the same question worked. Grep stays as the floor. Search sits on top of it.

For the broken links, we wrote a plain Python link graph that a nightly janitor rebuilds from the actual files. It gives the same answer every time you ask, which is exactly the property the graph lacked.

What I’d tell someone about to buy a graph

Write the kill criteria before the pilot, and include one that compares against the cheapest alternative. Run the extraction twice on the same input and diff the results. Read the issue tracker before the README. And test the update path, because the demo always shows a fresh build and you’ll live with the updates for years.

I still like the idea of a company knowledge graph. This one just wasn’t ready to be trusted by a team, and a graph that changes its mind every night is worse than no graph. More on how the rest of the brain fits together in the pillar post.

If you want search with sources over your own team’s documents, I build these in 14 days.