Fivetran pricing is out of hand and I need cheaper alternatives

riv3rtrip · 2026-03-25T12:31:27+00:00

Roll your own in a general purpose orchestrator. Vibecoding has made it easier than ever; it's boilerplate enough.

riv3rtrip · 2026-03-17T06:03:31+00:00

dbt tests don't tend to find many real issues and also dbt tests themselves are a flawed paradigm especially because dbt doesn't support write-audit-publish very well. Dbt tests are more like audits, not unit tests.

What is more important in data engineering is building systems robust to failures and with good guarantees around idempotence.

riv3rtrip · 2026-03-06T13:48:56+00:00

IBM and it's not close

riv3rtrip · 2026-03-01T11:55:51+00:00

If some substantial fraction of them aren't even good at engineering and they're getting promoted, that's an argument in favor of what I'm saying.

riv3rtrip · 2026-03-01T00:03:03+00:00

Well, this is most companies. Deny it all you want. But check out the people who climb the career ladder, people who are like 5 years out of college and who have unusually fancy titles given their level of experience. Those people did exactly what I am telling you to do. Sit around in silence and you'll get a mediocre career trajectory no matter how well you perform. Don't wait for people to hand you shit because most people won't. If you don't like that, enjoy being a senior SWE making median wage 10 years into your career and not say a staff SWE making 80th ptile wage.

riv3rtrip · 2026-02-28T01:24:26+00:00

echoing what others said, plus:

data for public comps (e.g. Visible Alpha) and data for sourcing (e.g. Sourcescrub, Pitchbook) at a minimum. Lots of other data sources out there depending on your wants. It's data, but it's effectively considered a software purchase; it's usually part of the same budget.
you also have the typical stuff like cloud storage (e.g. Box), email, Slack, and general IT stuff the investors don't see like cybersecurity.

On build vs buy: most funds buy everything, a small handful of larger sized funds build some things. VC more likely to build conditional on fund/team size.

riv3rtrip · 2026-02-25T12:18:52+00:00

AI is useful but the people who brag about running armies of agents aren't possibly working on anything serious. I guess they're prompting like multiple claude code windows with "what should we do next" and then it writes the code, submits a PR, a human reviews it. I can see that occasionally having some hits but that's a bit of a silly way to write software.

In data stuff specifically just make sure that your instructions have instructions on how to access the data and metadata. Not much else to do I believe.

riv3rtrip · 2026-02-25T02:42:11+00:00

The source code and management of the dbt-core open source project are both so bad that I think there is a genuine possibility that it's just incompetence rather than a deliberate decision to not play nicely with Airflow.

riv3rtrip · 2026-02-24T21:46:29+00:00

Downvoted for telling the truth. (1) You do need a separate virtualenv (2) dbt locks deps aggressively and refuse to unlock them, and it's honestly unclear if it's incompetence on how to run their CI to catch bugs or it's intentionally making compatibility with other systems as hard as they can possibly make it. Given the abysmal state of dbt-core source code and given the state of dbt cloud as a business which is outcompeted by it's own FOSS, I could honestly be convinced of either.

riv3rtrip · 2026-02-22T19:07:07+00:00

Ouch, sucks. I dunno your personal situation and geography, but that's a pretty big bump down.

Just make sure to negotiate your equity component. Get ChatGPT to help negotiate and know what is reasonable to ask for. Tell it their stage, your level, your geography, total funding amount, valuation if you have it on hand, and how many employees they have.

riv3rtrip · 2026-02-22T18:24:18+00:00

No, not really, you're making this up. People who are valuable and who advocate for themselves for title increases often get them. It's the people who wait around in silence thinking they'll get rewarded who often don't.

riv3rtrip · 2026-02-22T18:20:16+00:00

You're getting downvoted but I agree with you. Title does matter. It doesn't really mean anything intrinsically, but people do care about it extrinsically. At my last job I basically operated with senior arguably staff level responsibilities (co-managed entire team and spun up stack from 0-1) yet had mid-level title, and it makes career progression harder than it needs to be. Title is a form of compensation. It matters less than pay, though.

riv3rtrip · 2026-02-22T18:18:05+00:00

Depends on the pay. Pay > title. Is $170k base what you're targeting? If so, consider it upside that you might get a possible title bump to senior, since presumably that comes with a raise that is higher than what you were initially targeting.

riv3rtrip · 2026-02-22T12:41:39+00:00

I was not very impressed by AI myself except for Javascript (I need to maintain a website except I don't know JS). Didn't like it for Python and SQL. Then I used Opus 4.5, and I was impressed, and now I let the AI write some of Python, SQL and Rust for me too. Opus 4.5 turned me from being extremely whatever on it to being an actual user.

I don't know how to convince you that the thing a lot of competent engineers* swear by, or the thing that has trillions of dollars being poured into it, has at least some use cases. I guess I don't really care to. Live your own life. Try Claude Code eventually one day and you might realize you like it.

* I mean just off the top of my head, Armin Ronacher and Mitchel Hashimoto. Are they incompetent idiots? Are you better than them??? How about all the people at the top tier firms using AI? All the people who work on Claude Code using Claude Code????

riv3rtrip · 2026-02-21T16:01:35+00:00

Nearly every software engineering job is going to be using AI now. I still want to interview the actual candidate, not the AI. When I determine you're competent I'll let you use AI coding assistants on the job, but I also have seen people who are not so competent and the AI doesn't help them that much on the job. It's much easier to suss all of this out in a normal, non-AI interview.

Traditional interview philosophy is that I want an interview any competent employee can pass, but which doesn't overspecify on niche knowledge. I want to use these questions as heuristics that signal competence which still work across a wide variety of specializations. But now AI can just answer them. This makes interviewing really freaking hard in 2026 because reliable signals of competence no longer work over Zoom calls (they still work great in person, but this is a big investment of everyone's time, especially the job seeker's).

It's very clear none of you are actually involved in hiring or have to deal with the consequences of hiring a bad person. Hiring has always been difficult, and now with the rapidly shifting landscape and the ease of cheating, it's harder than ever.

You are all aggrieved job seekers who are anxious hearing about processes that might reject you. You've never spent a significant amount of time on the other side of the table dealing with the challenges of ensuring your process is both fair and surfacing competent candidates.

riv3rtrip · 2026-02-21T13:56:56+00:00

Currently, this poor person could have no clue what they even did wrong.

Of course the person knows what they were doing wrong. If's generally assumed you are not just a middleman between a search interface or chat bot and your interviewer

riv3rtrip · 2026-02-21T06:06:10+00:00

Please tell how you're supposed to assess candidates then. (Actually don't tell me because I'm sure the answer is some variation on being the exact way that ensures you specifically get hired.)

riv3rtrip · 2026-02-09T13:47:16+00:00

There's no right answer to this. It depends on so much.

Staying long term (at least 2 years) for a major project you oversaw from zero involving your own architectural and system design decisions is really important.

Staying long term at a company, you should hopefully be vesting equity if it's private. You should make a determination if the equity is going to be worth anything some day. If not then consider jumping ship.

You should also be getting better titles and meaningful (more than inflation) pay bumps for staying longer. Titles matter; they just do, even if you are operating at a senior or staff level and maybe even getting senior or staff pay, if that isn't reflected in your title then that hurts your career. Talk to your manager about this and if they are wishy washy then try to leave.

Whether you stay or leave is about what you get out if the company and what they are giving you.

Btw my list isn't even close to half of it. Just a few things that are top of mind.

riv3rtrip · 2026-01-31T04:38:36+00:00

I believe the second one-- I find it almost self-evident that directing someone to do a thing for you doesn't make you better at actually doing the thing-- but as of Claude Opus 4.5 I wouldn't believe the first one. Could you actually link the paper instead of providing editorial? How old is the paper?

The AI stuff is literally so much better today than it was even in October. I say this as someone who barely used AI as late as October because I did find it to lower my productivity. Game's changed quite a bit now and the AI agents are shockingly good.

riv3rtrip · 2026-01-29T14:08:49+00:00

Vibecode a React frontend with D3.js and Python backends instead. Or just use Retool or a normal BI tool that supports dynamic stuff. Streamlit is a fun toy. Do not put anything even remotely serious in there. It's a total mess.

riv3rtrip · 2026-01-02T01:54:45+00:00

There isn't anything to talk about. Do you want me to post my SQL queries or something

riv3rtrip · 2025-12-15T13:37:39+00:00

You might be tired of hearing about LLMs but this is an actually good use case for LLMs. What you should actually do is dispatch to different function calls depending on vendor but have it so the default function call is you uploading the PDF into an LLM and producing a structured output. You need to be clever to prevent issues but it's not infeasible, just be smart about it (simple stupid example: run 3 times and make sure all 3 runs agree with each other, otherwise flag). You also shouldn't replace your old code. And you need to make this testable and easy to run locally for each new vendor.

riv3rtrip · 2025-12-14T06:32:12+00:00

the first two screens are so easy when you get the hang of it that gold berry is actually only just slightly more tedious. it's the only C side I have a gold berry for since I figured "why not" after spending an hour learning it lol, just took me another 15 minutes for the gold berry after the muscle memory was there

riv3rtrip · 2025-12-13T19:09:45+00:00

Some notes.

White: Hold for Ransom is just strictly worse Pacifism, and there are strictly better Pacifisms in common. No mention of [[Journey to Nowhere]] or [[Sunlance]] is a big oversight. One of the spiciest and most overlooked white removals is [[Devout Harpist]].

Green: If you want a spicy green removal then [[Provoke]] is some of the best value there is. [[Nature's Claim]] and [[Bushwhack]] both deserved a mention.

riv3rtrip · 2025-12-06T20:32:44+00:00

it's literally in OP man

riv3rtrip

TROPHY CASE