# Memory Graphs Beat Vector Search for AI Project Context

---
title: "Why vector search keeps failing my AI projects (and what finally worked)"
published: true
tags: [ai, programming, devops, opensource]
---

Last week, an agent in a coding workflow did something that looked smart and was completely wrong.

It found a “similar” design doc in a vector store, pulled an outdated API shape from it, and confidently generated a migration against the wrong table. Nothing crashed immediately. Which made it worse.

The problem wasn’t that embeddings are bad. Vector search is great when you want **semantic similarity**.

The problem is that **project context is usually not a similarity problem**.

It’s a **relationship problem**.

If your agent needs to know:

- which service owns a table
- which PR changed the auth flow
- who approved a policy exception
- which file is blocked by another agent
- what task depends on what decision

…a memory graph usually beats vector search.

## The core issue

Vector search answers:

> “What looks like this?”

But project context often needs:

> “What is connected to this, who changed it, when, and what depends on it?”

Those are very different questions.

A vector DB might retrieve a chunk that *sounds* relevant. A graph can retrieve the exact chain of facts:

- `feature_flag -> introduced_by -> PR-184`
- `PR-184 -> modifies -> auth/middleware.ts`
- `auth/middleware.ts -> owned_by -> platform-team`
- `platform-team -> approved_exception -> policy-27`

That’s the kind of context agents can actually reason over.

## Why graphs work better for agent memory

Here’s the simplest way I explain it:

```text
Vector search:
[user asks about auth bug]
        |
        v
 find "similar" chunks
        |
        v
 maybe relevant text

Memory graph:
[user asks about auth bug]
        |
        v
   auth bug
    /   |   \
 owner  PR   policy
   |     |      |
 team  files  approval
        |
      tests
```

A graph gives you **structure**, not just proximity.

That matters because AI project memory usually has four kinds of data:

1. **Entities** — files, services, tickets, agents, users, PRs
2. **Events** — deployed, reviewed, failed, approved, locked
3. **Relationships** — depends on, owns, modified by, delegated to
4. **Time** — what was true *when* the agent made a decision

Vector search handles fuzzy recall well.  
Graphs handle **causality, dependency, and provenance**.

In practice, the best systems often use both:

- vectors for discovery
- graphs for truth

## A tiny example

Say your agent gets asked:

> “Can I safely change the billing webhook retry logic?”

With vector search, you might retrieve docs about billing, retries, and webhooks.

With a graph, you can answer the real question:

- which service owns webhook retries?
- what incidents were linked to this code path?
- which agent or human last changed it?
- is another agent currently working on the same files?
- does this flow require approval before deploy?

That’s much closer to how senior engineers think.

## A minimal memory graph in code

You don’t need a fancy stack to prove this out. Even an in-memory graph is enough to see the difference.

```bash
npm install graphology
```

```js
const Graph = require("graphology");

const graph = new Graph();

graph.addNode("billing/webhook.ts", { type: "file" });
graph.addNode("PR-184", { type: "pr" });
graph.addNode("incident-92", { type: "incident" });
graph.addNode("payments-team", { type: "team" });

graph.addEdge("PR-184", "billing/webhook.ts", { rel: "modifies" });
graph.addEdge("incident-92", "billing/webhook.ts", { rel: "caused_by" });
graph.addEdge("payments-team", "billing/webhook.ts", { rel: "owns" });

console.log(graph.neighbors("billing/webhook.ts"));
// [ 'PR-184', 'incident-92', 'payments-team' ]
```

That tiny example already gives an agent something vector search alone does not: **explicit connected context**.

Now imagine enriching it with timestamps, approvals, test runs, and agent session IDs.

## Where vector search still helps

This isn’t “replace your vector DB with a graph DB” advice.

Use vector search when you need:

- fuzzy recall across docs
- semantic matching for user questions
- retrieval from large unstructured corpora

Use graphs when you need:

- source of truth for dependencies
- change history and provenance
- multi-agent coordination
- policy-aware execution
- conflict detection

If your agent can take actions, not just answer questions, graph memory becomes much more important.

Because once an agent writes code, opens PRs, or touches infra, **“similar enough” stops being good enough**.

## The pattern that’s working for us

The most reliable setup we’ve seen looks like this:

1. **Ingest events** from code, tickets, tools, and agent sessions
2. **Build a graph** of entities + relationships
3. **Use vectors** only to find candidate nodes/documents
4. **Traverse the graph** to assemble decision context
5. **Log the path** the agent used so humans can audit it later

That last part matters a lot.

If an agent says, “I changed this file because it was similar to another chunk,” that’s weak.

If it says, “I changed this file because PR-184 introduced the retry logic, incident-92 was linked to it, and payments-team owns the service,” that’s something a reviewer can validate.

## Try it yourself

If you’re working on agent memory, a good next step is to tighten up the security and visibility around the systems feeding that memory.

- Want to check your MCP server? Try https://tools.authora.dev
- Run `npx @authora/agent-audit` to scan your codebase
- Add a verified badge to your agent: https://passport.authora.dev
- Check out https://github.com/authora-dev/awesome-agent-security for more resources

## Final thought

The mistake is treating project context like a search problem.

A lot of it is really a **state and relationships** problem.

Vectors are great at helping agents *find* things.  
Memory graphs are better at helping agents *understand what those things mean together*.

How are you handling long-lived context for AI agents right now: vectors, graphs, both, or something else?

-- Authora team

This post was created with AI assistance.