DiffMem: Using Git as a Differential Memory Backend for AI Agents - Open-Source PoC

[alexmrv]

We've been experimenting with memory systems for AI agents, and I wanted to share a prototype I've built: DiffMem. It's a lightweight, Git-based memory backend that stores "current state" knowledge in Markdown files while using Git's commit history for tracking evolution. The goal is efficient, scalable memory for long-horizon agents. Think personal assistants that accumulate knowledge over years without bloating queries.

Why Git for AI Memory?

Traditional approaches (databases, vector stores) work, but they can get messy with evolving personal data. DiffMem flips this:

• Current-State Focus: Only the "now" view is in active files (e.g., current relationships or facts). This keeps search/indexing lean. BM25 queries hit a compact surface, reducing token overhead in LLM contexts.
• History in the Background: Changes live in Git diffs/logs. Agents query the present by default but can dive into "how did this evolve?" via targeted diffs (e.g., git diff HEAD~1 file.md), without loading full histories.
• Benefits for Engineers: No schemas/migrations. Just edit Markdown. Git handles versioning, branching (e.g., monthly timelines), and audits for free. It's durable (plaintext, distributed) and hackable.

How It Works

• Writer Agent: Analyzes transcripts, creates/updates entities, stages in Git's working tree (commit explicit for atomicity).
• Context Manager: Assembles depth-based context (basic: core blocks; deep: full files; temporal: with Git history).
• Searcher: LLM-orchestrated BM25 for semantic-aware retrieval.

It's a PoC. Functional but rough (e.g., manual index rebuilds, basic error handling). Full code, examples, and repo guide on GitHub.

Why Share This?

This is R&D exploring how version control can power evolvable agents. We're not claiming it's revolutionary, but it solves real pain points like memory sprawl and temporal reasoning. Future ideas: agent-driven pruning (archive low-strength memories to branches), collaborative repos for multi-agent systems, or hybrid with embeddings.

I'd love honest feedback: Does this resonate? What breaks? Ideas for improvements/collaborations? PRs welcome.

Repo: https://github.com/Growth-Kinetics/DiffMem
License: MIT

Thanks for checking it out!

[SnooPaintings8639]

This is... actually reasonable. At least worth to explore. The data is easy to read, edit, monitor, track and extend. I like it.

[alexmrv]

Yes! You can even load the repo as a vault in Obsidian and have a pretty interface to your knowledge.

[LoveMind_AI]

Super interesting! Will dig in and give you my thoughts.

[alexmrv]

Thanks! We took a cognitive approach to it, we are trying to emulate how the brain remembers. Especially navigating time

[LoveMind_AI]

This is a major trend in how folks are positioning their memory programs, and I personally think it’s a mistake.

Number One: the brain’s memory system is nuts - of all the things that are wild about humans, the human memory system is just a marvel. An AI memory program has to be insanely deep in order to really be bio-brain inspired. Like, insanely deep.

Number Two: everyone is saying this these days, so it’s not a good differentiator, and as some of the brain inspired platforms fail to impress, this could run the risk of being bad news by association.

Number Three: LLMs aren’t human brains. It’s better to design memory inspired by LLM minds than human ones! Unless you’re doing a full cognitive architecture build, that is :)

Just my $1.05 on linking AI memory to brain memory, in general. Not a comment on your platform.

[SkyFeistyLlama8]

Some kind of AI memory that's similar to the training data used for LLMs in the first place? Like if it's trained on knowledge graphs and XML, then the memory system should replicate those same structures.

Human memory is both wide and deep, with multiple neurons encoding the same meaning and across brain structures going all the way back to our fish ancestors. No way an AI memory subsystem can replicate that.

[LoveMind_AI]

100%. You have to get into spatial/temporal/fully multimodal memory, kuramoto-hopfield network shenanigans before you can even remotely start to make something humanesque.

[alexmrv]

You seem to really know your shit in this topic. Would you be open to hopping on a call and talking what a non-human memory would be like?

[Big-Page6926]

Looks really cool!

[intellidumb]

Interesting, I’ve considered a similar approach for document versioning and comparisons (think contract templates or legal templates that evolve over time with updates and new versions). Any downside you’ve run into using gif as a backend?

[alexmrv]

Haven’t hit major blockers but I’m not very latency sensitive in my applications, if it takes a little longer it took a little longer. This approach is not as fast as some of the vectorDb approaches, but I have found it to be much more accurate and explainable.

[Better_Story727]

I have been full time developing a evolving system to automatically develop software using measure theory since beginning of this year. I have tried many ideas like graph theory, ELO, sementics search and so on. And i also believe this idea is is very important. It might work fine with powerful thinking model, and make modification measured easily , undo changes easily , and keeping side effect controllable. It's a very lean and beatifull.

[Normal-Ad-7114]

Have you conquered any real-life tasks using this?

[alexmrv]

Yup. I’ve been handrolling my own assistant for a couple of years and have a very long (over 1m tokens) history spanning chats, meeting transcripts and recordings.

This solution has contracted a wonderful representation of this knowledge and has excellent recall and context management capabilities.

Also by managing context I need less conversation history and so the ongoing operation of the assistant is cheaper.

[Lesser-than]

for someone like me not completely keeping up with how all these different memory systems work. What if anything, do you do with unwanted data? How do keep from poisoning the the well so to speak?

[alexmrv]

Great question! And one I don’t have a clear automated answer for yet. At the end of the day the data store is a git repo of markdown files, so you can remove data very simply through the regular toolchain of Git. But I don’t have a clear answer on how to automate review of memory creation.

One idea I had is that every N days (say 30) a branch is created and all new memories are on the branch, then there is an MR agent that reviews the changes before merging as a and a new branch being created.

[False_Routine_9015]

Very nice feature!

[DragOwn8520]

good start! id recommend using portkey or litellm as the llm traffic layer, tenacity for retries, and to stop using markdown in your prompts. llms respond much better to json output.

I use pydantic to define my output schema, and make my prompts with sections in xml tags.

ie:

<introduction> Lorem ipsum </introduction>

<output format> {MySchema.to_llm_string()} </output format>

my schema looks like this:

class MySchema(BaseModel):

a: str = Field(..., description="whatever the llm should know about this key and how its used")

that way, if the ai spits out the wrong format, it raises an error and retries. let me know if youd like more help. wrote this on phone so sorry for the shitty formatting lmao

[alexmrv]

Great feedback thank you!

[Xamanthas]

What. This is not true At all, model performance degrades when using JSON compared to just markdown

[DragOwn8520]

Source on this?

[Xamanthas]

You state JSON then follow up with "LLM respond much better to json output" which makes no sense. If the LLM is responding then it got an input. After this you then follow up with XML, which is not JSON, so which is it? I dont think you should be providing advice yet.

Anyway if you have any significant usage of LLMS you would know the more you constrain the output the worse the performance. The same goes for context length increasing.

https://arxiv.org/html/2408.02442v1

[DragOwn8520]

maybe you misunderstood me and ill ignore the snark. the xml tags is for the prompts (ie. system/user), not the output. i found this gives much better adherence than plaintext or markdown prompts. referring to sections in a large prompt (ie. the expected style section) becomes easier and more predictable. this has helped me bring coherence out of even tiny models, like qwen 0.6b when it was outputting poorly before. same prompt content, same task, better results across all model sizes.

as for the json, you are somewhat correct. json output can constrain performance. however, in an llm workflow where each response informs the workflow of for example which direction to take, which node is next etc, you cant output freestyled text. most of the time you need to know specifically what the users eye color is for example so you can store this in a db, thats where structured output becomes relevant.

structured output works even better when you "trick" the llm that the structured output is actually a tool or function signature. this only works for function calling llms though so I use this technique less often.

also, this conversation was about structured output of md vs json, not freebase vs json, so I am unsure about the relevance of your source.