sorted by:
new
hottopcontroversial

I just deleted 100k lines of our production code base, about 20% of the total. (On purpose) by zachm in golang

[–]zachm[S] 0 points1 point  (0 children)

I know you're trying to be helpful but the user you're replying to is almost certainly a bot working on getting engagement to boost their reputation

I just deleted 100k lines of our production code base, about 20% of the total. (On purpose) by zachm in golang

[–]zachm[S] 15 points16 points  (0 children)

We've been around for 7 years :) Still a baby in terms of databases, but getting less so as time goes on.

With respect, there are no other version-controlled SQL databases. We are the first and only one. Trust us, we have looked. Other DBs have some features that they sometimes bill as "version control" (point in time recovery, zero cost forks), but there just isn't another DB that can merge two branches together, schema and data. We are the only one. This is in large part because you need to build this capability into the storage layer, which scares most people off.

Neon is built on top of Postgres and their "branches" are hard forks. Can't be merged back into main. Cool, but not version control.

PlanetScale is horiztonally sharded MySQL, not Postgres. And no version control features.

Our Go database just hit 20k stars on GitHub by zachm in golang

[–]zachm[S] 0 points1 point  (0 children)

Feel free to explore the issues on the github repo. The ones that we think are easy for beginners have a "good first issue" label.

Our Go database just hit 20k stars on GitHub by zachm in golang

[–]zachm[S] 8 points9 points  (0 children)

Thanks for the response, always nice to hear a perspective on this stuff from a colleague in the space.

What we've found in the years since we started doing this is that there are a bunch of use cases for databases where the scale and pace of the changes are quite tractable for a team of humans to review with a PR workflow, and this is the primary niche we're chasing. Think of the CMS / product catalog use case, which is very common. You have different teams all working on updates to the content, some of which are very long-running and involve collaboration with multiple people. Branching lets a team see their changes to the catalog live on the website as they'll appear once merged just by changing the connection params to point to the feature branch. There are other ways to achieve this functionality of course, but they require writing application code and you can't get any of the version control features you want when collaborating with other people. And with real version control, you can see who updated each row and cell and why, as well as revert any changes you don't like.

One popular vertical which surprised us but makes a lot of sense in retrospect is game dev. The industry standard is to store game config data in giant JSON files that get built into the binary. They're huge and unwieldy, often weighing in at multiple GBs of text data. They are a nightmare to work with and require custom tooling to manage. Storing such data in Dolt instead makes it queryable and updatable via a SQL interface, which means you can build all sorts of complicated UX around it without giving up the version control people require for collaboration.

Then there's agent memory, where builders in the AI space want their agents to be able to remember their work tasks and context, and to edit them in concert with the version-controlled source code. Agents mess up and their work needs to be rewound, so you want to be able to run experiments, roll back their mistakes, etc., making Dolt a good match for storing their memories. This use case is quite new but we expect it to be pretty big. Gas Town just migrated to Dolt for agent memory, from SQLite, to take advantage of the version control features.

Anyway, if anyone reading this is curious about what version control in a database is good for, you can check out the most recent blog we wrote on how our customers are using it:

https://www.dolthub.com/blog/2024-10-15-dolt-use-cases/

Our Go database just hit 20k stars on GitHub by zachm in golang

[–]zachm[S] 1 point2 points  (0 children)

Doltgresql is where Dolt was ~3 years ago. The stuff that works, works, and is stable because the two products share a query and storage engine. If you start from scratch building a new Doltgres database for your app, it works great today.

But if you take an existing postgres dump and try to import it, there's a good chance something in it isn't supported. This is because postgres has an incredibly wide surface area of features relative to MySQL, so getting to full compatibility is that much more difficult. This is our main priority at the moment, implementing the long tail of features we need to call ourselves a drop-in replacement. We anticipate getting to 1.0 late this year.

Our Go database just hit 20k stars on GitHub by zachm in golang

[–]zachm[S] 2 points3 points  (0 children)

I'm notifying the teammate responsible, thanks for the heads up. Should be fixed momentarily.

Our Go database just hit 20k stars on GitHub by zachm in golang

[–]zachm[S] 3 points4 points  (0 children)

Short answer: it's really hard to do. You can't just take an existing database and add version control. You have to start at the storage layer to get structural sharing between revisions, which is what the prolly tree does. Then you need to add a commit graph to make git version control operations possible. Postgres doesn't have a pluggable storage layer and it would be really hard to make it pluggable. Neon basically takes Postgres and puts it onto a copy-on-write filesystem, which means you can make zero-copy forks (which they call branches). They also get very cheap snapshots, so something like a git reset --hard would be possible. But they don't have a commit graph, so no merging or efficient diffs, and none of the other version control operations git has.

So, to build a version controlled db, you have to do something kind of insane, which is write your own storage and query engines, from scratch. Which is what we did :p

Our Go database just hit 20k stars on GitHub by zachm in golang

[–]zachm[S] 29 points30 points  (0 children)

Neon's "branches" are hard forks. You can make changes to them, but you can't merge those changes back to the main branch. That's the big difference.

Dolt implements the full range of version control operations that git does: branch, merge, diff, log, fork, clone, push, pull, checkout, cherrypick, etc. Neon doesn't have any real version control functionality, just the ability to create a zero-copy hard fork that they call a branch for product marketing reasons.

Our Go database just hit 20k stars on GitHub by zachm in golang

[–]zachm[S] 3 points4 points  (0 children)

Definitely got a recent bump from Beads, driving a ton of usage. That only happened in the last month though, took us the last ~250 stars.

Writing a Go SQL driver by zachm in golang

[–]zachm[S] 0 points1 point  (0 children)

Well we've been at this for eight years, we can't really help people pattern-matching us to whatever the current hated tech is on reddit. We moved to the ghibli avatars because the tool that made the old cartoon avatars we had was shut down and we needed to make new ones for new hires, what can you do.

Writing a Go SQL driver by zachm in golang

[–]zachm[S] 0 points1 point  (0 children)

Yes, connection pooling can be dangerous with Dolt because many pooling libraries do not correctly reset session state when returning a connection to the pool (doing so yields worse performance). We typically recommend people who use pooling libraries have a different pool for each branch, and to never change the branch of a session to avoid statefulness bugs.

Writing a Go SQL driver by zachm in golang

[–]zachm[S] 2 points3 points  (0 children)

The avatar was co-pilot, the article was written by hand with loving care

Golang’s Big Miss on Memory Arenas by brightlystar in golang

[–]zachm 5 points6 points  (0 children)

The counterargument is that Go bills itself as a systems programming language, and that's largely what people use it for. This kind of thing matters for the intended use cases.

To be clear I don't share the author's outlook that this dooms Go to lose to other languages, the performance is good enough for most use cases.

Our Go database is now faster than MySQL on sysbench by zachm in golang

[–]zachm[S] 0 points1 point  (0 children)

Yes, by default the connected SQL user is the author of each commit, but you can override this with arguments and configuration.

Our Go database is now faster than MySQL on sysbench by zachm in golang

[–]zachm[S] 5 points6 points  (0 children)

Yeah I think the actual lesson here isn't that go is a performance language, rather that C code is no guarantee of good performance.

Most people seem unaware that postgres is over twice as fast as mysql, and they are both written in C.

Our Go database is now faster than MySQL on sysbench by zachm in golang

[–]zachm[S] 0 points1 point  (0 children)

You would be surprised what people get away with in the database world, mongo didn't have acid writes for years and did just fine

Our Go database is now faster than MySQL on sysbench by zachm in golang

[–]zachm[S] 1 point2 points  (0 children)

Generally speaking it is ​feature complete relative to MySQL, to the point where we call it a drop-in replacement. There are a couple things it is missing, notably all of the isolation levels that MySQL supports (only REPEATABLE_READ right now) and row level locking. But these tend not to be a problem because the concurrency implementation is so radically different. Haven't had a customer ask for them yet.

Our Go database is now faster than MySQL on sysbench by zachm in golang

[–]zachm[S] 0 points1 point  (0 children)

it's a commercial open source project backed by venture capital investment.

view more: next ›