I wanted memory that lasts across Claude sessions, and that I could explain after the fact when an agent surfaced the wrong thing.

The reflex when you say "agent memory" is to reach for a vector store. Drop in an embedding model, chunk the corpus, cosine-nearest-neighbor at query time. That answers a question I wasn't actually asking. I didn't want "things similar to what we're doing now." I wanted "what should the agent do right now, given that it's working on this codebase, in this category of decision, and was told something a week ago that still applies." Similarity is a fuzzy shadow of applicability. Sometimes they overlap. Often they don't.

So I built a small MCP server. It speaks the protocol over stdio, the Claude Code harness spawns it at session start, and it persists into a SQLite file in my home directory. Each row is a typed learning with a problem and a solution, a category, some tags, and a project tag. The interesting twist: when a learning is wrong, you don't overwrite it. You write a new row and link the old one. The old version stays in the database for forensics. The agent only sees the current truth.

I've been using it for a couple of months and it has become the most boring piece of infrastructure I rely on. The boring part is the point.

The pipeline

There are three ways a learning ends up in the database. Most of them go through an LLM that just understood the conversation, which turns out to matter.

The first source is a sub-agent that fires after every agent turn. It looks at the conversation that just happened and decides whether anything novel was said. Did the user correct an approach? State a preference? Resolve a bug after multiple attempts? Establish a practice that should outlive the turn? If yes, it calls one of the storage tools itself. If not, it does nothing. The prompt is a few lines telling it what counts as reusable knowledge and what to skip. The actual judgment is done by the same model that just sat through the conversation. That's the cheapest classifier I'm ever going to find.

The second source is a filesystem scan that runs when a session ends. Claude's harness writes per-project memory notes to disk in markdown with YAML frontmatter. The scan walks those files, maps the frontmatter type into one of the categories, and ingests anything new. It's incremental: a small scan log records the modification time per file, so re-scans are nearly free.

The third source is a bridge to another MCP server, claude-mem, which stores its own conversation observations. Its data is only accessible through its own tools, so the bridge tool exists as a place to dump the JSON result and have it dedup-and-merge into the same store. Currently the scan log shows around 820 observations processed across all of that.

All three feed one schema. One database. One query surface.

What's in a row

The row shape matters more than I expected.

Each learning has a problem field and a separate solution field. Most agent memory systems store a single content blob, which is a missed opportunity. Splitting them turns out to mirror how engineers actually think about reusable knowledge: this is the situation, this is the response. The classic case-based-reasoning split. The problem field also carries the bulk of the retrieval signal, because the words used to describe a situation tend to overlap with the words an agent will use when it's facing a similar one later.

Beyond that, each row has a category, free-form tags, a project name, a timestamp, and an optional pointer to a successor row. The category is one of six: practice (what good looks like), bug-fix (a concrete failure and its resolution), tool-pref (sticky preferences about libraries and CLIs), pattern (a reusable design move), workflow (a multi-step process rule), and env (config, deployment, machine notes). Six is small enough that the agent can mentally pre-filter at query time. Twenty would be paralysis.

The taxonomy is also the join point for the two import paths. Different sources speak different vocabularies, and the import tools have small lookup tables that translate.

Querying it

There are about a dozen tools on the server. Most of them are obvious: store, list, query, update, delete, and a handful of scanners that pull from the three ingestion sources I described. Three are worth describing in their own sentences.

query_learnings is the one the agent calls most. Full-text search over title, problem, solution, and tags, with structured predicates on top: category, supersession state, optional project filter. The ranking is the same keyword-relevance algorithm Postgres and Elasticsearch use under the hood. There's a quiet fallback path: when the agent passes a raw natural-language query with characters the full-text parser can't tokenize, the code catches the parse error and degrades to a substring search with the same structured filters. The agent doesn't write search-engine syntax, it writes how a developer thinks. The server absorbs the mismatch.

supersede_learning is the unusual one. It doesn't mutate. When a learning gets corrected, this tool inserts a brand new row with the new content and points the old row at the new one. Every retrieval filters out rows that have been superseded, so the old version becomes invisible to the agent but recoverable by ID. This is the same idea as event sourcing or temporal tables: state changes are append-only, the current state is derived. The reason I care about this for agent memory is debuggability. When the agent retrieves the wrong thing and applies it, I want to know exactly what it thought the right answer was at the moment of failure. In a store that overwrites, that information is gone. In this one, it's a SELECT.

scan_session is the one I'd rebuild from scratch if I started over. It walks a raw conversation transcript and looks for the substring "error" or "fix" as a coarse bug-fix mining heuristic. It works the way a regex always works, which is to say sometimes. The Stop-hook sub-agent is a strictly better signal source for the same job, because it has semantic understanding instead of grep.

Why structured, not similarity

The argument for skipping the vector store is shorter than I expected when I started writing it down.

Vector retrieval gives you "looks like." This system needs "applicable to." The predicates that determine applicability, things like the category of decision, the project the work is happening in, whether a learning has been superseded, are not properties that emerge from text. They're declared at write time, by an agent that just sat through the conversation and knows. Filtering on declared fields gives you a tiny, dense, applicable result set. Cosine similarity gives you a wider, noisier set that then has to be filtered in prompt context.

Beyond that, the entire debugging surface is a SQL shell. If the agent retrieved the wrong learning, I can run a query against the database, see exactly which candidates were considered, see what was filtered out, and supersede the offender. With a vector store, the equivalent of "why did it retrieve this?" turns into "cosine similarity ranked it third, here are the 768 dimensions," which is operationally unrecoverable in the moment. The boring SQLite-shell debuggability is most of why this design works.

There's a corpus-size argument underneath both of those. Vector retrieval earns its keep when the corpus is too large for structured-query coverage to be tractable, and when the queries have no exploitable structure. Agent memory is neither of those things. The corpus is small. The queries have structure because the agent itself writes them.

What I'd change

A few things this implementation doesn't do, in rough order of value.

Hybrid retrieval is the obvious next step. A small local embedding model over titles and problems, running in parallel with the keyword search, blended via reciprocal rank fusion. Structured filters apply to both legs. The point isn't to replace the structured layer but to catch the long tail where the agent's query vocabulary doesn't match the stored vocabulary. Vectors as a complement, not a substitute.

Auto-categorization with a confidence score would help with the cases where a practice gets miscategorized as a pattern, or vice versa. Right now the Stop-hook agent picks, and it's usually right. A dedicated classifier with a confidence threshold could flag the ambiguous ones for review instead of just guessing.

A pre-store hook that checks for near-duplicates and surfaces them would close a real gap. Today the agent has to know to call the supersede tool instead of the store tool. That requires it to recognize that new content contradicts an existing learning, which is easy when the old text is in the prompt window and hard when the contradiction is two hops away. A pre-store nudge would help.

A decay term on the query path would push stale entries down without deleting them. A bug-fix from a year ago about a deprecated library shouldn't outrank a fresh one. The half-life should probably vary by category.

And the one I keep meaning to build: a retrieval eval harness. Labeled query-to-applicable-row pairs collected from real sessions, plus a script that reports precision and recall at a few k values. The day hybrid retrieval lands is the day this becomes necessary. Without it I won't know whether the blend is actually better than pure keyword search.

Closing

The system is running. The database lives in a SQLite file in my home directory. Last write was earlier this week, when the Stop hook captured a preference about lockfile precedence. Sixty rows of typed evidence accrued, and I lean on them every day.

Most agent memory systems reach for a vector store reflexively. Build the structured layer first. If you ever need vectors, layer them on top of fields you can actually query.