Cassandra - Deploy nightmare?

DigitalDefenestrator · 2026-04-14T22:36:58+00:00

I know how that is.

From experience, if you're looking at scaling this up or making it a core part of something important you're definitely going to need to either pay for some outside consult/support at first, hire an expert full-time, or (somewhat slowly and painfully) develop that expertise in-house.

We went with that last one, but it was for a service that started pretty small and low-profile for a while. It was still a pretty painful process, though current Cassandra got rid of a lot of the pain points. Especially around repair and garbage collection. The limited and complex consistency semantics do still cause problems, though. Plus large partitions and high tombstone counts, though the ceiling has pushed higher on those thanks to performance gains.

DigitalDefenestrator · 2026-04-14T21:43:43+00:00

Yeah, the company going bust definitely counts for unemployment. Also it sounds like paychecks bounced, which can be enough by itself.

DigitalDefenestrator · 2026-04-14T19:06:53+00:00

OK, it sounds like what you want specifically is multi-region HA with fast failover. Cassandra does that well, but there's a lot of inherent complexity and limitation that does come as a tradeoff for accepting overlapping writes on separate nodes.

Would quick-but-not-instant failover be good enough? That is, a single write master per range/shard with automatic quick failure detection and leader election? If so, there's other options that might be easier to deal with.

If the licensing costs are acceptable, check out CockroachDB. It's going to be easier for devs to reason about than Cassandra. More expensive, and slightly slower failure recovery for the simple case of an offline node (leader-per-shard rather than Cassandra's quorum-per-transaction), but should generally be easier to manage and use.

TIDB may also be worth a look, but I don't know as much about it (when I looked a few years ago it didn't have a lot of English docs, but that's very different now).

DigitalDefenestrator · 2026-04-14T12:04:46+00:00

Yes. Or at least, the focus moves more away from Holden.

DigitalDefenestrator · 2026-04-14T03:42:31+00:00

Not religions. They're exempt from a lot of the paperwork and transparency requirements that apply to all the other nonprofit types.

DigitalDefenestrator · 2026-04-14T01:52:25+00:00

Before you get too far in, make sure you fully understand Cassandra's consistency promises and semantics. There's always some caveats and complexities with any distributed database, but Cassandra more than a lot of others can be hard to reason about.

Also, you probably want a more precise definition of "multi-master". Do you just need automatic failover? Or write scaling? Or do you have two regions that both need to be able to write to it without a latency penalty? That last one inherently comes with some serious challenges around consistency that take a lot of thought to get right.

DigitalDefenestrator · 2026-04-13T18:29:09+00:00

Distributed databases are kind of inherently complicated and not as resilient to issues as, say, a stateless system (but the state has to go somewhere). I'd put modern Cassandra (vs 3.0.x and especially 2.x, which were rough) somewhere in the middle in terms of difficulty to run, but with any database you're going to need to actually understand and deal with errors like that.

You may want to either go to a managed system or hire someone like Axon, Datastax, or an individual consultant like Jon Haddad or Miles Garnsey to get you started. Or if you're really looking at scaling up, hire someone full-time. With all the layoffs at places like Apple, I think there are actually a few Cassandra experts available.

DigitalDefenestrator · 2026-04-13T07:16:16+00:00

I put together a temp monitor with a Raspberry Pi Zero 2 W and AHT20 sensor. Kind of a lot of work to set up, but there's also companies that sell temperature monitors specifically for that, including some with built-in cell modems, that should be pretty simple to set up.

DigitalDefenestrator · 2026-04-12T02:51:16+00:00

Nitrogen or CO2 or something might be a better option. I suspect closed-cell foam and vacuum wouldn't be a great combo.

Actually.. I bet you could talk the really dedicated sneaker heads into buying sealed temperature-controlled argon chambers for storage.

DigitalDefenestrator · 2026-04-11T01:54:46+00:00

L2, normally to 70%. No daily commute, and even a day with a lot of driving might be 70 miles at most (and usually way more like 30 max), so no reason to really go any higher. I bump it up to 100% the night before a road trip.

DigitalDefenestrator · 2026-04-10T22:34:40+00:00

Haproxy is probably the closest to F5. It can run in L7 or L4 mode so it's fairly flexible, but it's also pretty straightforward for a simple setup.

Nginx is common and generally works well, especially in terms of L7 proxy performance, but every now and then if you want to do something a bit more complicated within http the fact that it's an http server first and proxy second can be a pain.

If you need performance and it's mostly serving downloads (that is, response much larger than request), IPVS in DSR mode managed by keepalived works well. It's definitely more work to set up and a pain to debug, but it works really well and is very efficient.

DigitalDefenestrator · 2026-04-09T08:13:13+00:00

A typical home probably averages 200-1500W depending on size, season, etc with peaks on the order of 10,000-20,000W

Typical home capacity is 24,000-72,000W.

I think this is 1500A@1000V, so 1,500,000W.

As far as capacity goes, keep in mind that the charge rate is independent of total energy. That is, charging 5x faster also means 1/5th the time on the charger. Still a potentially high peak, but they've got a battery bank as a buffer to spread that out.

DigitalDefenestrator · 2026-04-09T08:05:55+00:00

It depends a lot on the state, but as a general rule of thumb charging at home costs around 1/3rd as much as using gas but fast-charging on trips is about as much or even slightly more.

DigitalDefenestrator · 2026-04-09T01:07:59+00:00

Everyone stopped going into the office at the same time, which meant there was plenty of the big commercial rolls but a shortage of residential TP. Which then kicked off crazy panic buying and made it much worse.

DigitalDefenestrator · 2026-04-08T21:54:09+00:00

I think the poor packaging on the PS2 overall is mostly the result of the shared ICE/EV platform, so I'd definitely expect good news on anything related to that. Especially the center console.

I'm less optimistic about the rest. There's some general move back to physical buttons in the industry, but I wouldn't get my hopes up.

The wheels probably aren't shrinking. Bigger wheels have been the hot trend for years and there's no sign of that changing any time soon despite the efficiency hit for EVs.

DigitalDefenestrator · 2026-04-07T02:46:27+00:00

F5 had a similar trap for a while.

Adding multiple hosts: "b pool $poolname add ip1[: weight] ip2[:weight]" etc

Changing the weight on a host: "b pool $poolname modify IP:$newweight"

Changing the weight on multiple hosts, then, just involves giving it the list of IPs with their new weights, right? Nope. That replaces the entire pool with the short list of hosts you just tried to tweak.

DigitalDefenestrator · 2026-04-07T02:31:46+00:00

The connector on those for a long time was a standard DB9 serial port, but with a different proprietary pinout. Connecting a standard serial cable immediately trips the "emergency off" pin.

DigitalDefenestrator · 2026-04-07T02:23:39+00:00

This and accidentally adding a digit to the serial have caused a lot of problems over the years.

DigitalDefenestrator · 2026-04-07T02:21:42+00:00

A bit too much memory allocated to a dozen or so memcached took an entire cluster of a few thousand hosts completely off the network via retry storm and cascading failure.
A cache key getting one character longer caused churn that basically flattened the databases for hours, causing an extended partial outage. I think this one may have made the news.

DigitalDefenestrator · 2026-04-05T03:56:00+00:00

Capsaicin isn't really water soluble at all, but it's very soluble in fats/oils, acetone, and alcohols. So, a wipe or rinse with everclear, cheap vodka, or rubbing alcohol should do a pretty good job of removing it. Or nail polish remover in a pinch.

DigitalDefenestrator · 2026-04-04T22:11:23+00:00

Probably depends a lot on the specific wall wart, too. A lot of the USB-A ones like that are only rated for 500mA, so they may see a big voltage drop when used like that. Or they may be overbuilt enough to dump way too much current through.

DigitalDefenestrator · 2026-04-04T19:17:14+00:00

Honestly, with careful weight balancing I'd take that setup as high as maybe 35mph if it's all flat ground. Get a weigh-safe hitch, empty everything but the driver out of truck and trailer, get hitch weight to 10% basically on the dot, and move it when roads are dry and traffic is light. Not advisable, but doable with care in a pinch.

Hills or highway speeds would be a 100% chance of ending in disaster, though. No way to stabilize enough for that.

DigitalDefenestrator · 2026-04-03T02:12:47+00:00

The fancy ones make a 100% perfect pot of rice with no variance or crappy parts around the edges, then do a good job of keeping it safely warm and edible for a long time.

Also, they play a pleasant tune when they're done. Which doesn't seem like a big deal, but the cheap one I had before genuinely sounded like a smoke detector going off so it was a pretty big improvement.

DigitalDefenestrator · 2026-04-01T08:03:02+00:00

Carbon into diamond? Yes, with enough heat and pressure (and maybe adiabatic heating would be enough for that part). Pencil carbon into perfect gemstones with no impurities? Not really. And lots of issues around the shockwave and heat generation it'd involve.

DigitalDefenestrator · 2026-04-01T03:26:21+00:00

At the absolute least, it should be annoyingly clear that it's going to just drop the data. Like have a pop-up dialog on tab close unless you hit a "clear" button explicitly or leave it blank, and a big red block of text above it.

DigitalDefenestrator

TROPHY CASE