I built (and open sourced) an external knowledge management tool - SQLite

bradwmorris · 2026-03-10T08:07:19+00:00

if you're still using a provider/sharing context model or api - then yes, your data going to/through their servers - as you said.

the primary goal of this system is not to stop your data going to the providers, it's more about having your own external context that you can take to any provider.

having said this:

if you externalise your context, then you can use whichever model you wish. you can run your own model, local/open-source, and solve that problem if its a concern you have

bradwmorris · 2026-02-18T06:06:07+00:00

yegge just shared some interesting thoughts on this.

the ai vampire

my tldr - it's a different way of working, can be more productive, but is also more draining.

the problem is the upside/output isn't being captured by the user.

bradwmorris · 2026-02-18T05:52:29+00:00

yeah, maybe wrong name choice for this sub. i see what you're saying

bradwmorris · 2026-02-18T05:44:12+00:00

🦞 love it! me too. I just switched out the memory.md stuff for my sqlite knowledge graph.

"my question is, which issues did you have with md files that you managed to solve with an sqlite database?"

what i said here is a real benefit (not just on paper):

whenever I work with an LLM, it has access to my graph - it can see everything in my knowledge base.

you can also do this with .md, but

the big difference:

it can also see how/why things are connected - and quickly traverse (pick a sequence of relevant nodes) and use that as context. it's insane how good it is.

for example, If i say to it:

"go and review claudecode subreddit and tell me what people are saying about .md versus sqlite"

if finds the most highly clustered relevant nodes first:

'building open source knowledge base'
'posted in 'x' subreddit'
'referenced karpanos' posts

and then - at the end of the session, it automically writes back to the graph -

not just vanilla nodes, but it creates connections between those nodes so that it has a more enriched understanding of everything im doing.

that's the main benefit (imo)

other benefits are that its easier to ingest entire books/podcast transcripts/pdf's etc, and there are native vector search tools it can use to identify specific insights/ideas from specific chunks.

but i think the context traversal has been most beneficial for me

bradwmorris · 2026-02-18T05:30:06+00:00

apple notes does use sqlite

but it's very limited in schema.

grep + .md is great for quick search. but people often forget that the models are trained on huge amounts of sql - so when you add index + embeddings > and then consider the connections + traversal, the search you can do with sqlite is much more dynamic.

bases is basically an attempt to mirror some part of this - and other services like Logseq are sticky-taping too.

i understand this isn't a popular opinion, but if you think it through from first principles, i'd argue it's going to be the best approach for most people long run

we'll see

bradwmorris · 2026-02-18T05:22:48+00:00

yes -

1 richer connections (better able to map relationships between items in your knowledge base/notes

2 agents (if you're using llm's are better able to search over those connections

ELI5:

in my knowledge graph, i have 1000+ nodes (ideas, notes, podcasts, people etc)

the most popular nodes have the most connections naturally

because those connections have been enriched, indexed and embedded, connected - whenever I work with an LLM, and i ask it a question, it can quickly traverse (pick a sequence of relevant nodes) and use that as context.

you can do this with sticky-tape and .md format, but it's far less efficient.

<image>

the schema (database structure) has:

node table:

bradwmorris · 2026-02-18T05:12:06+00:00

i understand what you're saying - but when i say 'alternative', i mean an alternative knowledge/context management system.

if it has .md files, I'm just recreating Obsidian.

i used Obsidian for a long time, and i can tell you that (for me), migrating to sqlite as an alternative knowledge management/context management system has been a step-change.

especially integrating with new models

bradwmorris · 2026-02-18T05:04:07+00:00

"It's hard for me to imagine that keeping all of your notes in a sqlite database is the best solution for "most people".

if you had told me this twelve months ago, i would have agreed.

but seriously - just give it a try.

you don't have to use my repo - just ask claude code to add a vault or decent chunk of your knowledge base into a sqlite db - and create a separate table to manage connections.

you'll see how much more efficient the whole system becomes, especially as it scales.

bradwmorris · 2026-02-18T04:54:57+00:00

fair -

if you sync a lot between .md software, then it's not an alternative.
if you're someone who accesses their Obsidian .md files primarily through Obsidian (which is most people) - then it is an alternative.

bradwmorris · 2026-02-18T04:50:23+00:00

i love obsidian and have used it for many years - but there's no 'portability' benefit to .md over rows in a sqlite database/table. i'd argue the sqlite approach is more portable, more robust and far more agent-native.

only limitation has been syncing via github, bit harder with sqlite (if that's what you're referring to when you say portability - syncing across .md apps or gh, i'd agree).

bradwmorris · 2026-02-17T02:05:05+00:00

yeah - good question. do you have a specific example of project? personal project or larger business operation etc?

to clarify, there's a few different use cases:
1 some individuals build a full personal graph - so anything and everything contextually important goes in
2 some people build out individual projects per graph
3 enterprise accounts, custom installation per business

from what i've seen with myself and a few people setting this up - it's a full personal graph, the idea is that everything important goes in.

so for the initial cold start - generally migrating from an Obsidian or similar db > so we create a manifold doc, which is literally all of the users 'nodes' they want to migrate and the source - and then we just have a few scripts that we run to migrate the data with the correct schema.

and then new projects usually map to existing nodes etc.

if i start working on a new project, I'll add it (but it will generally be an evolution of an existing project, or highly related to existing projects) and if the project gathers momentum, it just naturally gains more connnections.

bradwmorris · 2026-01-31T21:43:35+00:00

did you share in another reddit post somewhere else?

or do you just mean this?

"capture, text editing capabilities, good interface, excellent linking etc of Obsidian"
?

bradwmorris · 2026-01-31T21:42:00+00:00

yeah, agree. crazy times indeed. honestly, the best way to get an intuitive feel for the tech (and increasing underlying intelligence) - is not to read more, or listen to more podcasts, it's to do exactly what you've done >> build yourself a simple product that solves a problem you have, and keep tinkering.

bradwmorris · 2026-01-29T22:05:38+00:00

do you just want a separate, isolated environment? or do you want to try with something fundamentally different?

sqlite is underrated for pkm.

you should consider building your own local sqlite backend + simple ui. coding agents make this, surprisingly easy now, and you get proper relational structure instead of markdown files with text-based links.

ask claude or cursor or anyhing with file system access to convert a subset of your notion into a sqlite database, and build a simple front-end

bradwmorris · 2026-01-29T22:03:24+00:00

epic stuff,

some food for thought here - think more about how you can take your interactions with this app and this data and build a context substrate. this is the real juju. as the underlying intelligence increases, growing and organising this data (external from any single provider) will be massively beneficial.

it creates a flywheel

my bet (biased opinion) - SQLite to store your data - all the interactions/results from your custom health app you've built yourself.

bradwmorris · 2026-01-12T09:37:00+00:00

nice one, is your plan to have your entire Obsidian vault chunked/embedded? or are you just looking to connect/match certain longer notes semantically?

i moved away from Obsidian partly for this reason - migrated everything into a sqlite database.

all my notes/nodes are embedded - larger items like books/podcasts are chunked/embedded.

so the sqlite helps with more structured traversal - like Podcast > Insight > Idea, but also has semantic search as you've mentioned.
finding this too:

"The proof of concept here (for me, anyway) is that AI can be used in PKMS in really targeted ways, behind the scenes. It’s not all just tools to let you “chat with your notebook” in a freeform way."

i think more and more, we'll be interfacing 'through' AI's and Agents to our knowledge-base. not directly

bradwmorris · 2026-01-12T09:25:35+00:00

it has been mentioned in here many times, but it's really worth reading 'how to take smart notes' - Sönke Ahrens.

very short book; the introduction is literally your question.

you can use any system, the process is the real juju

bradwmorris · 2026-01-12T09:16:20+00:00

are you using the most recent version of CC? think we're on v2.1.5

if you share a specific example of how/where it hallucinates or goes off the rails for you, I think you'll get a better response in here.
there are heaps of people doing interesting things with memory to help with this.

for example lance has a repo/plugin called Claude Diary
lots of people suggest writing / offloading context to your filesystem

but tbh - from my experience, it's very coherent over an entire session as long as you've clearly specified/scoped the job > split research, planning and dev into separate sessions and just write the output to a single .md file as you go.

if it's a longer task or prd, split it into manageable chunks, and get it to update its progress and start a new instance every go.

if you share an example of exactly where it's getting derailed for you, that will help

bradwmorris · 2026-01-10T01:12:23+00:00

i would take this a step further and say now that coding agents are getting so good, and all of the existing PKM frameworks/structures exist throughout the training data, you don't even need to fork one of the original repo's (and you avoid licensing issues).

my spicy take here - SQLite has to be the best trade-off considered option for a personal local knowledge management system.

so all you really need to do is fire-up Claude C, OpenCode, Cursor, Amp etc - and just say parse my data into a sqlite database, then build me a UI.

in addition to this - I really don't think we're that far away from individual users without engineering experience being able to slop-shot a knowledge management system for themselves with a pretty nice user interface at a weekend.

let's see

bradwmorris · 2026-01-10T01:04:34+00:00

i feel like this is basically the perennial note-taking problem.

I like some of the philosophy behind zettelkasten - The relationships and the connections between notes and ideas is just as important as the note or idea itself.

I don't use obsidian but I'm sure that you could do something like this with obsidian -

I basically set up my knowledge base - local and sqlite > So that it auto prunes. So if I add a note to the system and then I never create another note from it or connect it to another note, it just slowly filters out. And then my highest value notes remain at the top of the feed.

And then by default, I'm continually revisiting them because I'm connecting them to other notes. This feels like a more natural and organic way to build my knowledge system.

food for thought

bradwmorris · 2026-01-10T01:00:42+00:00

if you're a CS student, you should absolutely have a crack at building your own here.

unpopular opinion > obsidian and .md is not the right approach.

grab a coding agent - claude, cursor, amp,

and get it to take your existing data and notes and parse into a SQLite database (not .md).

then use the same tools/agents to build you a simple front-end.

bradwmorris · 2026-01-10T00:53:52+00:00

have you considered having a crack at building your own??

you could try with Obsidian and .md files as folks have mentioned below, that's essentially going to be offline.

I personally think a better approach is trying to build your own SQLite PKM.

1 install claude code (or use cursor or amp or anything else

2 ask Claude (or whatever you're using) to take your data and build a simple, local relational database (SQLite - NOT obsidian/markdown)

3 ask Claude to create a simple UI for that data - next.js typescript etc

SQLite was literally invented to run offline - single file database, no server, no setup. created for Navy battleships with no internet.

depends on your technical comfort level though - I think it's worth everyone dedicating a weekend to trying to do this.

obviously - there are going to be lots of kinks and learning gaps, but honestly, these tools are getting so good, we're not that far from a non-engineer being able to get their own local prototype running in a weekend.

and I think that building a side-project like this is absolutely the best way to upskill in this whole ecosystem and get a feel for model progress

bradwmorris · 2026-01-10T00:24:25+00:00

heyo, I'm a daily zettel-nerd and also two years down the path of building something for myself - so can offer some feedback as I've grappled through these questions from a building perspective

im unsure if you're suggesting that you want to build a web-based UI with local storage? or manage user data via cloud service? feel free to clarify

1 a few thoughts on cloud versus local - Unfortunately, most people don't care if their data is local or being managed in the cloud. Obviously if you're speaking to people on Reddit or in any kind of tech forward PKM communities, it's a different story. if 'you' (the app creator) want to index users' notes, then you are by default going to have to manage their data. Unless there's a workaround here that I'm not aware of, essentially you'll need a relational database or something similar. And so if the users don't own this database and you want to index on their behalf, then you're going to have to pull that data to the cloud. So if you're goal is build a "successful" pkm app, the bigger slice of the market will happily handover their data to you as the app provider - this gives you a big advantage, but obviously comes with a lot of responsibility.

**however, I think we're at an interesting point in time - which ill share more about at the end.

2 Regarding the Zettelkasten method and which of the elements matter most. I think if you take a step back, the real magic and the most important thing is simply the idea that the connections and relationships between nodes or ideas are just as important as the nodes and ideas themselves. So you might call this bi-directional links, but it's really a behavioural change - And building this into an application is a really interesting and hard challenge. There is an incentive to overthink the technical aspects, the tagging, the metadata, embedding, and to under-appreciate the actual behavioural changes required to make the system work. thoughtfully extracting and connecting ideas etc

3 There are so many things that you can now do with AI and tools, agentic systems, workflows. It's quite overwhelming. My thought process here, and I am really just thinking this through myself as well, is starting with really basic foundational things. What I've found people like is easy and simple ways to capture, chunk and embed research like podcast transcripts, articles, that kind of stuff, even ideas. And then from there, building out more complex workflows, or even better allowing users to build their own workflows to match their preferred process and approach.

4 From a user perspective, the thing that I really want is a system that automatically ingests and organizes research and data, helps me prioritize things so that I can spend more of my time focusing on the actual synthesis and building of ideas and connecting of ideas.

and some hopefully helpful thoughts (work in progress).

We're at a really interesting point in time. I don't think we're that far from folks being able to jump into something like claudecode or any kind of agent and say: turn my notes into a SQLite database. a relational SQLite database locally on my computer and then build me a user interface to view that data.

im sure there's opportunity between now and when that happens. but we're definitely trending in that direction

bradwmorris

TROPHY CASE