Memory Graphs Beat Vector Search for AI Project Context

---

title: "Why vector search keeps failing my AI projects (and what finally worked)" published: true

tags: [ai, programming, devops, opensource]

Last week, an agent in a coding workflow did something that looked smart and was completely wrong.

It found a “similar” design doc in a vector store, pulled an outdated API shape from it, and confidently generated a migration against the wrong table. Nothing crashed immediately. Which made it worse.

The problem wasn’t that embeddings are bad. Vector search is great when you want semantic similarity.

The problem is that project context is usually not a similarity problem.

It’s a relationship problem.

If your agent needs to know:

• which service owns a table
• which PR changed the auth flow
• who approved a policy exception
• which file is blocked by another agent
• what task depends on what decision

…a memory graph usually beats vector search.

The core issue

Vector search answers:

“What looks like this?”

But project context often needs:

“What is connected to this, who changed it, when, and what depends on it?”

Those are very different questions.

A vector DB might retrieve a chunk that sounds relevant. A graph can retrieve the exact chain of facts:

• feature_flag -> introduced_by -> PR-184
• PR-184 -> modifies -> auth/middleware.ts
• auth/middleware.ts -> owned_by -> platform-team
• platform-team -> approved_exception -> policy-27

That’s the kind of context agents can actually reason over.

Why graphs work better for agent memory

Here’s the simplest way I explain it:

Vector search:
[user asks about auth bug]
        |
        v
 find "similar" chunks
        |
        v
 maybe relevant text

Memory graph:
[user asks about auth bug]
        |
        v
   auth bug
    /   |   \
 owner  PR   policy
   |     |      |
 team  files  approval
        |
      tests

A graph gives you structure, not just proximity.

That matters because AI project memory usually has four kinds of data:

1. Entities — files, services, tickets, agents, users, PRs
2. Events — deployed, reviewed, failed, approved, locked
3. Relationships — depends on, owns, modified by, delegated to
4. Time — what was true when the agent made a decision

Vector search handles fuzzy recall well.
Graphs handle causality, dependency, and provenance.

In practice, the best systems often use both:

• vectors for discovery
• graphs for truth

A tiny example

Say your agent gets asked:

“Can I safely change the billing webhook retry logic?”

With vector search, you might retrieve docs about billing, retries, and webhooks.

With a graph, you can answer the real question:

• which service owns webhook retries?
• what incidents were linked to this code path?
• which agent or human last changed it?
• is another agent currently working on the same files?
• does this flow require approval before deploy?

That’s much closer to how senior engineers think.

A minimal memory graph in code

You don’t need a fancy stack to prove this out. Even an in-memory graph is enough to see the difference.

npm install graphology

const Graph = require("graphology");

const graph = new Graph();

graph.addNode("billing/webhook.ts", { type: "file" });
graph.addNode("PR-184", { type: "pr" });
graph.addNode("incident-92", { type: "incident" });
graph.addNode("payments-team", { type: "team" });

graph.addEdge("PR-184", "billing/webhook.ts", { rel: "modifies" });
graph.addEdge("incident-92", "billing/webhook.ts", { rel: "caused_by" });
graph.addEdge("payments-team", "billing/webhook.ts", { rel: "owns" });

console.log(graph.neighbors("billing/webhook.ts"));
// [ 'PR-184', 'incident-92', 'payments-team' ]

That tiny example already gives an agent something vector search alone does not: explicit connected context.

Now imagine enriching it with timestamps, approvals, test runs, and agent session IDs.

Where vector search still helps

This isn’t “replace your vector DB with a graph DB” advice.

Use vector search when you need:

• fuzzy recall across docs
• semantic matching for user questions
• retrieval from large unstructured corpora

Use graphs when you need:

• source of truth for dependencies
• change history and provenance
• multi-agent coordination
• policy-aware execution
• conflict detection

If your agent can take actions, not just answer questions, graph memory becomes much more important.

Because once an agent writes code, opens PRs, or touches infra, “similar enough” stops being good enough.

The pattern that’s working for us

The most reliable setup we’ve seen looks like this:

1. Ingest events from code, tickets, tools, and agent sessions
2. Build a graph of entities + relationships
3. Use vectors only to find candidate nodes/documents
4. Traverse the graph to assemble decision context
5. Log the path the agent used so humans can audit it later

That last part matters a lot.

If an agent says, “I changed this file because it was similar to another chunk,” that’s weak.

If it says, “I changed this file because PR-184 introduced the retry logic, incident-92 was linked to it, and payments-team owns the service,” that’s something a reviewer can validate.

Try it yourself

If you’re working on agent memory, a good next step is to tighten up the security and visibility around the systems feeding that memory.

• Want to check your MCP server? Try https://tools.authora.dev
• Run npx @authora/agent-audit to scan your codebase
• Add a verified badge to your agent: https://passport.authora.dev
• Check out https://github.com/authora-dev/awesome-agent-security for more resources

Final thought

The mistake is treating project context like a search problem.

A lot of it is really a state and relationships problem.

Vectors are great at helping agents find things.
Memory graphs are better at helping agents understand what those things mean together.

How are you handling long-lived context for AI agents right now: vectors, graphs, both, or something else?

-- Authora team

This post was created with AI assistance.