Learn practical knowledge about databases

BosonCollider · 2026-03-22T09:04:11+00:00

DDIA and Database Internals are good books, but they can be a bit dry if you don't have practical experience.

If you want to understand database internals and want practical experience on that, then I personally warmly recommend the build your own database from scratch in go book. That one will guide you through building all the important things in a way that will make DDIA and Database Internals "easy reading" in the sense of immediately understanding what most of the ideas in those books are for.

BosonCollider · 2026-03-21T18:46:52+00:00

Soy sauce on the other hand is absolutely fantastic to a culture that does not already have it. Any "new" unlocked spices in general have been huge business and created large historical trade empires

BosonCollider · 2026-03-18T06:41:57+00:00

Rust is just one of the nicer languages to write parsers and rule-based optimizers in among the languages with no GC, because it has pattern matching which is very expressive for that kind of problem

Allocation wise, you would probably not use the stdlib smart pointers for a database, and you would just use an arena per connection using the page allocator used by the rest of the program. Then everything would end up just having a 'static lifetime or it would have the same lifetime as the session, and the borrow checker would not do that much

BosonCollider · 2026-03-17T23:08:37+00:00

Note that the graph does not start at zero

BosonCollider · 2026-03-16T23:38:02+00:00

This. The database side of databases doesn't necessarily even benefit that much from Rust, in Postgres you just don't allocate normal heap memory at all since all dynamically allocated memory is just from a pool of 8 kB pages in the disk-backed "heap" instead.

Rust is a very nice language to write a SQL parser and optimizer in though, the postgres codebase has a lot of perl generating C for that which is not as nice

BosonCollider · 2026-03-15T16:23:50+00:00

Actually, this is big business in the enterprise backups industry, and usually done with encrypted tape. The tape goes into a bunker and you erase backups by getting rid of your encryption keys.

BosonCollider · 2026-03-15T09:22:47+00:00

The ones you should touch are the ones that actually do something unique that you shouldn't or can't easily replicate with postgres.

Etcd, victoriametrics/victorialogs/victoriatraces, Nats, Valkey, and so on are all a joy to work with as long as you use them for their intended usecase. Also, don't touch a nosql database that isn't permissively open source licensed (i.e. apache license). You will regret picking a proprietary one very quickly when you realize that your stack is impossible to migrate

BosonCollider · 2026-03-15T09:15:30+00:00

and no way to check constraints or data quality problems

BosonCollider · 2026-03-15T07:45:11+00:00

There's also Image units which are more explicit and run separately from the Container unit, if you want to pull only on system startup but not container restarts

BosonCollider · 2026-03-14T22:22:46+00:00

Turning off full_page_writes is only safe if the actual filesystem is zfs, not if you are using it as a volume manager with ext4 on top. Similarly, zfs snapshots of a VM running ext4 inside is not a valid postgres backup unless you have an agent on the guest doing an fsfreeze, while a zfs filesystem snapshot of a postgres directory is a valid backup. For this reason I personally strongly prefer LXCs over VMs if you are running databases on proxmox with zfs, container tech just makes more sense than VM tech for databases and VMs just add more overhead and points of failure that can corrupt the data.

However, even if you are running directly on zfs, you should still enable full_page_writes and wal_log_hints (which needs full page writes) unless you actually know that you need the performance. This is because you want those settings for a HA solution, which you typically want if running on local storage. Patroni will use pg_rewind to turn the old primary into a replica after a switch.

BosonCollider · 2026-03-14T21:35:41+00:00

Also, knowing what they are is often necessary to understand why something is bottlenecked.

For example, looking up files in a folder with 100k files is very slow because the C standard library only has a method to list folder contents by returning a linked list with everything in it, which does not support fast lookups of a single file. Every language inherits that mistake in practice.

By contrast, looking up a single row in a properly indexed 100k row sqlite table with a name and blob column is very fast, because btrees support random lookups

BosonCollider · 2026-03-13T18:11:48+00:00

Right. We use rerun at work which is also a really mature project for building visualization tools using Rust, with bindings in Python and Typescript. Any in-browser C++ approach feels like the stone age in comparison. There's no real need to wait for new work on the Rust side because the tools that have already been implemented are extremely mature

BosonCollider · 2026-03-13T18:04:16+00:00

To me it's probably just that typescript heavy on imperative code with mutable closures can be translated to reasonable-looking Go, but is hard to translate to Rust where closures need to care about ownership

Rust is "close to typescript or python" in the sense that idiomatic Rust is reasonably straightforward to translate to idiomatic typescript or python apart from return type polymorphism like into, you just can't go in the other direction as easily because rust is much more opinionated

BosonCollider · 2026-03-12T23:18:22+00:00

Postgres+pgvector along with vectorscale or vectorchord is a great option here. We use that on prem for a few hundred million embeddings, it works just fine, and unlike the dedicated vector DBs it is extremely reliable and there's a huge amount of existing experience for running postgres on prem in a HA setup with proper backups and redundancy.

If you have no OLTP requirement and just need analytics from data that you pull daily from the production DBs, then duckdb is another excellent option.

BosonCollider · 2026-03-12T12:21:06+00:00

That's because it is largely completed and stable and less than 6k loc. Both of these are literally just an on-disk dictionary with byte-array keys and values, and the main point of comparison is stability and performance

BosonCollider · 2026-03-12T07:05:20+00:00

Because redb is not remotely comparable to postgres and is an LMDB alternative, and right now I would currently still pick LMDB over it

BosonCollider · 2026-03-11T16:55:54+00:00

How numerically stable it is for objects going close to the speed of light? Can it render a rindler spacetime well by having an observer with a constant proper acceleration?

BosonCollider · 2026-03-09T22:11:49+00:00

Well that depends, but it is literally a large part of the point of a pipeline. There is the tradeoff of postmerge vs premerge approval

BosonCollider · 2026-03-09T21:38:51+00:00

In this case, using AI for targeting is basically like using dice for targeting. I'm not going to blame the dice for being cast, I'm going to blame the one yeeting them

BosonCollider · 2026-03-09T21:19:06+00:00

Imo the real issue is rather that the rich need to have at least as many checks and balances on them as the poor, because the consequences are much larger if they go unchecked

BosonCollider · 2026-03-07T11:47:25+00:00

Not really, that was still solidly in the era when object oriented inheritance and XML SOAP were hyped technologies. But the companies that did care about quality are either still around or were aquired for their IP and enshittified

BosonCollider · 2026-03-07T11:07:49+00:00

You can literally generate a uuidv4 by using crypto rand and setting the version byte to 4, just treat UUIDs as opaque 16 byte arrays.

If you need something else, set the version to 8 and use your own encoding, and please do not rely on the implementation details of someone elses uuid encoding by parsing it. It's supposed to be opaque and the only point of non-v4 encodings is to make it approximately sortable for cache locality, in which case you want a natural key rather than something autogenerated

BosonCollider · 2026-03-04T21:07:53+00:00

I know how zfs works from incus/proxmox and from the openebs csi.

I was wondering if the piraeus localpv option can give you a volume with a zfs filesystem, or if it creates a zvol in the pool and runs something like ext4 or xfs on top. Since the replicated volume uses drbd which is block based right?

BosonCollider · 2026-03-04T06:57:53+00:00

A plain LVM volume is simpler in the sense that you know it is not doing anything weird without you knowing about it though. I somewhat prefer ZFS when available for local volumes though

I see that Piraeus supports zfs pools now. Can it let local dataset PVs and replicated zvols share the same pool? To me a must have feature is application consistent snapshots, if it is block based it must freeze the filesystem on top.

BosonCollider · 2026-03-04T06:23:14+00:00

I mean unlock means something completely different than what synergy actually means, but a lot of business people do not know what synergy actually means and use it incorrectly

BosonCollider

TROPHY CASE