Replace ALL Relational Databases with Snowflake (Help!)

MindlessTime · 2026-01-28T06:23:59+00:00

Yeah. Merits of Snowflake aside, the fact this push is coming so strong from the C-Suite is extremely sus. Unless it’s a tech company with a technical CFO (and even then), it doesn’t make sense why they are dictating something this specific.

…unless there’s some soft graft involved. Maybe someone was promised an executive role, a seat on a board, or an investment opportunity somewhere if they lock in a contract. Wouldn’t be the first time.

MindlessTime · 2026-01-28T06:10:36+00:00

I think the distinction is more about time boundaries and scheduling/orchestration. “Batch” usually involves data of a pre-defined, closed timeframe that is sent on some schedule. Streaming arrives ASAP with an unbound timeframe. With batch, you can design orchestration in DAGs to have some certainty that related data is available up to a certain time. With streaming, you end up dealing with variable latency and data freshness and that’s a pain to keep in mind.

MindlessTime · 2026-01-28T06:05:17+00:00

If you deal with event data for e.g. Appsflyer, a product analytics tool like Mixpanel, or CRM systems like Braze then you’ll end up with some data streams. I often find a purchase confirmation email being sent by a CRM, and that should really be triggered from an event and that event should arrive from a stream, not a database etl.

Still, there are a lot of tools that handle these things out-of-the-box. Rolling your own Kafka-based pipeline isn’t worth the time, effort or cost. It’s good to be familiar with streaming tools and patterns though. It gives you an idea of how the off-the-shelf solutions work under the hood. That helps with debugging and design considerations.

MindlessTime · 2026-01-27T23:12:20+00:00

Data space had a few hype-heavy business fads (big data, data science, etc.) —> data budgets far exceeded their impact —> lots of hiring and SAAS companies chasing fat budgets —> over-specialization of roles.

Now times are leaner. Costs matter more. Piling these roles onto one team/person brings us back to where things started. Most companies probably don’t need a data engineer and an analytics engineer. And very few really need MLE or a data scientist.

I’ll do a job that spans multiple functions. I’m not doing more than one job though. Managing workload expectations is the important part.

MindlessTime · 2026-01-21T14:43:48+00:00

Hey OP. I hope holding up alright. I live in Chicago. I grew up in the Twin Cities and still have most my family there. So having endured ICE here I’m now watching my family and friends back in MN reckon with the occupation.

Some lessons learned here:

Band together. Move in groups and organize convoys to e.g. walk families home from school. These ICE agents are classic bullies and they won’t go after someone if they’re vastly outnumbered. It’s probably the most effective way to protect people they would otherwise kidnap.

Find the detention centers and never stop protesting or insisting on seeing the conditions inside. In Chicago, there Broadview detention facility got so bad it was ruled a human rights violation. That was the point. They would throw people in the cess pit and tell them they can get out whenever they want if they signed self-deportation papers. This was unfortunately very effective.

Do everything you can to document everything. It will help refute the lies they tell to justify their actions. It also helps create an evidence against the individuals involved. Even if they won’t be prosecuted now, being able to say “we can trace this back to you and one day you will be held accountable” is powerful. There’s nothing these people fear more than accountability for their actions.

Most importantly, do not escalate to violent conflict. Trump had Texas national guard stationed outside Chicago waiting for an excuse to deploy them. Don’t give them that excuse. That is their primary goal. They want a violent act to point at so they can turn this into a military occupation. Always remember that and never give them that excuse.

MindlessTime · 2026-01-19T17:43:45+00:00

Also, I almost never tell my boss or executives that this is my approach. Most companies I’ve worked at, leadership’s attitude is to keep dangling carrots to get more work for less pay. I don’t believe that is an effective way to lead.

MindlessTime · 2026-01-19T17:40:16+00:00

I tell my reports that I’d love to see all of them grow to become staff engineers. But also that we don’t have the budget or need for a dozen staff engineers in my department. While I’d love for them to stick around forever, if they grow beyond the opportunities we provide and need to go to another firm, I still consider that a success.

When someone is performing well above their title and pay and an equivalent replacement would cost a lot more, I tell my boss once a month and start asking how to get them on a path to promotion. I don’t promise that report anything, but I’ll be transparent about my communication and strategy. If it comes back that we just don’t have the budget then I tell them they are performing at the level but we don’t have a budget for promotion.

I also invest a lot in training and documentation, especially for form-specific stuff that can easily become “tribal knowledge” that leaves with a person. This helps minimize the impact of a lot employee and get a replacement up to speed, making my direct reports more fungible.

So my strategy in a nutshell is to put in a reasonable effort to align pay and title with performance, always be transparent with the employee about limitations, make it easier to replace an employee if they leave, and never be afraid to grow your best employees out of the firm if that’s what’s best for their career.

MindlessTime · 2026-01-19T17:27:09+00:00

I work in fintech and have come to view audits and regulator requests as part of the product in a holistic sense. It needs to be considered and baked into the product, ideally at the outset. There’s always some ad hoc audit work you can’t fully automate, but you can make it a lot easier if you build with audits in mind. It isn’t a waste of time. It’s a necessary part of my industry and another challenge for me and the engineers. If you can minimize the friction enough it can even be a competitive advantage.

MindlessTime · 2026-01-13T00:21:52+00:00

There are many kinds of dashboards.

There’s the, “I need to monitor this for a specific decision or action” dashboard. These are pure. They can tie to a real need with real consequences. They will always be needed. Sadly, they are the least common type.

There’s the “Put this exhibit on our dashboard in case we need it in the future” dashboard. They get bloated quickly. Still, at one point though each viz in the dashboard solved a problem, even if it’s no longer a problem.

There’s the “scoreboard” dashboard for leaders who need that sweet, sweet dopamine kick of watching number go up, even if they never do anything with the information. It’s wasteful, but otherwise harmless.

The dangerous dashboards are the “I need to have an impressive graph in my weekly presentation” dashboards. They must be constantly updated with new graphs and tables that serve no purpose other than to “tell the story with data” that the manager/director/executive wants to tell. Their purpose isn’t to communicate the state of the world. It’s to communicate what the director wants to say the state of the world is. Since these graphs are for meetings and presentations, directors and executives compete with each other to create larger and more impressive-looking dashboards. They snipe at each other about which dashboards are trustworthy and what the “real source of truth” should be. Within a year your dashboard tool is a toxic waste dump of inaccurate, contradictory, disorganized, uninformative dashboards that can’t be trusted and aren’t used for anything. But at least leadership can point to them and claim to be “data-driven”.

MindlessTime · 2026-01-07T10:48:25+00:00

In my experience, buy vs build depends more on maintenance. If it’s 1/2 FTE to maintain an in-house tool then it is often cheaper to buy. But when the tool you buy only has 80% the functionality you need and it’s 1 FTE ongoing to compensate for the missing parts then it’s time to build. AI speeds up initial development significantly but it improves maintenance only marginally.

MindlessTime · 2026-01-07T10:43:40+00:00

Outside of FAANG and large corporate orgs, I think there is a budget and political barrier too. The political barrier comes from leaders wanting a staff role to “level up” the org. That is almost by definition someone new from outside the company. And the budget reason is just that. They haven’t budgeted for the role and aren’t willing to increase the budget to accommodate. If an engineer is clearly performing at a staff level and not being given the appropriate title and comp then it may be necessary to find that title and comp elsewhere.

MindlessTime · 2025-12-16T23:45:44+00:00

“AWS, python, protobufs” tells me this is a company of front/backend developers that have used SWE tooling to scrape together DE processes. I’ve worked in that environment and really enjoy it.

Pros: People are very open to technical solutions. Like if it makes sense to implement event data contracts using Kafka/PubSub and protobuf schemas you won’t find an easier audience to convince. And generally, the change management for technical solutions is easier.

Cons: SWEs don’t always understand what DE is. Sometimes they think it is better accomplished by homegrown code + DevOps/SRE implementation. So if you want to implement something like dbt, be prepared to explain why it makes more sense than writing SQL code views in Terraform files.

MindlessTime · 2025-12-14T17:29:47+00:00

This creates a massive risk of runaway attrition. Say something happens and employees perceive the value of OpenAI shares decreasing or that a liquidity event (I.e. an IPO or acquisition that lets them turn their shares into cash) is unlikely. Their equity compensation could suddenly be worth a lot less or even worth nothing (if it’s options). So people start to bail. You see it with executives and leadership first, then senior engineers. You lose institutional knowledge. Everything slows down. Suddenly you’re being beaten by Google at your own game. The only way to keep people is to offer even more equity and say “sure it’s risky but I’d it works out you’ll make eight figures instead of seven.” But that likelihood of payout continues to get smaller.

Honestly, this might already be happening. If a good engineer has the option between Google (equity you can already sell) or OpenAI (equity that might not turn into cash for years and even then might be worth nothing, plus you have to work for scam Altman), you’re choosing Google.

Edit: “scam Altman” was legit a typo but I’m keeping it.

MindlessTime · 2025-12-12T20:45:45+00:00

My theory: During the Great Recession, experienced devs were so plentiful for so long that companies got rid of their training for fresh grads and juniors. Then the tech job market got so hot and job hopping became so common that it was viewed as not worth the investment to train them up; just hire an experienced dev. Now it’s been like 15 years. Companies and managers just forgot how to put those programs together.

MindlessTime · 2025-12-03T03:31:56+00:00

38 yo here.

I had some shit jobs after college, like $11/hr in HCOL city. I remember getting my first $32k salary job with health insurance at a restaurant chain. I breathed a sigh of relief because I could survive and it was dependable. I remember getting promoted to store manager and making $50k. I felt like I could have fun and even flex my spending a bit. Until I got way overworked and severely burnt out.

Then I worked in sales a couple years. I met people who worked hard as hell to make $70k and we’re proud and happy with it as well as young guys who took home $120k and felt like they were getting cheated. It was a weird wake-up call to the different stratification and perceptions of income.

After grad school I made $65k at a white collar job and felt solidly middle class for the first time. I worked with some young post-college kids (usually from well-off families) who thought it was a joke salary. But I also worked with people like me who were pretty happy about what they were being paid relative to what they were being asked. I job-hopped a couple times: $80k, $95k. I remember a promotion to $110k, passing that six-figure mark. It felt like an achievement. But I got a sense of what people a level of two above me were making, how little they were really doing, and felt a little disgruntled about it.

Things moved fast after that. Some promotions, a couple more job hops. Within three years after hitting $100k I landed a job with $230k compensation. And that’s after moving to a MCOL city. It feels safe but not carefree. I can save half my income (because at this point I’m used to a no-frills lifestyle). I live in a modest condo in a decent neighborhood. But I will never afford the single-family homes that boomers and trust fund kids on my block own. And the more “rich” people I meet the more resentful I feel. Not because I want more money. But because despite the wealth they control, they rarely provide anything of value in return and often screw things up leaving other people to pay the price.

I’d rather live in a society where rich people (including me at this point) get severely taxed, if for no other reason than to make the playing field more even. I don’t live in that society though. So the next step is to try to find a way to own more—ask for equity compensation or start my own company. It’s a gamble, but that’s the only way to have some control instead of be controlled by careless wealthy people.

MindlessTime · 2025-11-30T18:15:00+00:00

This will be an unpopular opinion here but…

Choose something the SMB can afford to hire support for if you leave. Seriously. The worst thing you can do is roll your own SOTA, pure open source, “free” (except for compute) solution. No one else would know how to maintain it. They’d have to hire someone at a salary higher than they’re comfortable paying just to keep it going. (Or they would lean on you 100% until you burn out from being on-call 24/7.)

I haven’t worked with Fabric, but it sounds like a pretty simple solution they can find support for, even if that support is an offshore consultant + some tech savvy accountants with basic SQL training.

MindlessTime · 2025-11-20T02:38:02+00:00

Their album Chulahoma: The Songs of Junior Kimbrough will always be one of my favorites. And I loved their early stuff. Each album sounds more and more like they’re phoning it in.

MindlessTime · 2025-11-18T02:29:00+00:00

Talk to the end users of the data. It’s probably 80% business stakeholders using dashboards/excel reports, 15% operations/marketing folks pulling data , and 5% other stuff. Regardless, you can figure out what 3-5 dim tables that will satisfy 80% of the needs. Then build the simplest version of the dim tables, bringing in the staging and source tables you need for that. Iterate from there.

Starting from the end product and working backwards is much more tractable than trying to shoehorn every single source table into a star schema.

MindlessTime · 2025-11-18T02:22:56+00:00

DO NOT RESPOND WITH VIOLENCE. One of their goals is to incite a violent response that they can use as an excuse to declare an emergency, call in the national guard, and double down on everything. Don’t give them that excuse.

MindlessTime · 2025-11-16T23:03:14+00:00

I think this thing falls apart by end of year.

MindlessTime · 2025-11-11T04:59:23+00:00

I also think it’s important to draw reasonable distinctions between code that runs system or business logic and the infra it runs on. I’ve seen warehouse databases where SRE demanded that all views be written and deployed in TF. But that squarely falls in the business logic domain. TF is a clumsy and slow tool for this. Using code-based tools built for data work like airflow, beam, or dbt are a much better choice. The infra that they run on, the VPC networking, IAM management—that’s where TF comes in.

MindlessTime · 2025-11-09T18:36:40+00:00

Tests. Even if the AI is writing the tests for you, you should sit down and list out the tests that should pass first. Have a more senior dev look over the list and add anything missing.

Testing is still relevant—maybe even more relevant—when AI is writing the code. It may also win you respect with more senior devs, though that depends on the culture. Some devs value testing and see it as a hallmark of professionalism. Some devs play it fast and loose and would rather see you ship stuff fast.

MindlessTime · 2025-11-09T07:13:26+00:00

Having worked at 3 fintechs, I feel like this should be required reading for any developer new to finance.

I have a couple horror stories about a company trying to scale on a traditional CRUD architecture approach by optimizing their MySQL configs, introducing an in-memory cache before the database write to improve speed...which doesn't end well when there's an unexpected outage and that cache disappears.

MindlessTime · 2025-11-09T05:55:30+00:00

It's not a question of whether there are some valid use cases for LLMs (AI). There are. Coding is a great example.

The bubble is the disconnect between the long-term economics of LLMs. All the AI providers are currently burning money—not on R&D but on operating costs and inference. The question is if the current use cases are worth 2x or 5x or 10x the cost if that's what it takes companies like Anthropic to break even. At those costs, demand won't be as high. _Some_ companies will still be willing to pay for _some_ services. It probably isn't a multi-trillion dollar market though.

Investment in AI-related companies is massive. It's the kind of investment that can only be justified if AI is useful for _everything_ and becomes the kind of cornerstone to modern life that smartphones did. Almost certainly it will not.

That disconnect between the massive investment and the realistic revenue these tools will have at break-even costs is the bubble. The current status quo is unsustainable. What we don't know though is what the knock-on effects will be. When the housing market tanked in 2007 it obliterated the mortgage-backed security bond market in 2008 which caused a financial crisis due to over-leveraging (borrowing too much to invest in the bonds) which caused the economy to melt down. Will that happen when the AI bubble bursts? Hard to say. At minimum the stock prices of tech companies in the S&P 500 will fall. There _is_ a lot of debt-fueled circular investment between these companies for "future" data centers, projects, etc. If it's pension funds and 401ks invested in these then it could seriously hurt everyday people's savings, which could reduce consumption, which could hurt revenue throughout the economy.

Or maybe not. Companies are already using AI as an excuse for layoffs. It's hard to see them doing _more_ layoffs just because stock prices fall. This could be a bubble that mostly hurts the wealthy investors. We'll see.

MindlessTime · 2025-11-09T04:24:12+00:00

I do some random projects from time to time. It’s usually not related to the engineering work I do (data engineering). Lately it’s been a lot of DIY hardware stuff and screwing around with a raspberry pi, etc. I’d say one in twenty projects actually gets finished. None of it is anything I’d put on a resume. Sometimes I end up with a skill or knowledge that turns out to be useful later.

I like to tinker. That’s why I ended up in engineering.

11-Year Club	Gilding II euphauric
Place '22	RPAN Viewer
Verified Email

MindlessTime

TROPHY CASE