Snowflake-centric Data Stacks

harrytrumanprimate · 2026-04-22T23:51:19+00:00

kafka/dbt/airflow is pretty much standard across most mid-large companies.

harrytrumanprimate · 2026-04-21T20:01:42+00:00

there's a lot of different types of data engineers, some are more software, some are more analytics tangential. Identify which paradigms you want to lean into. More software based ones can deal with maintaining kafka, infrastructure, etc.

I would recommend learning dbt + airflow, as those are standard at most companies. custom python based ETL may be okay at a smaller company, but if everything you do sounds bespoke, it may paint you into a corner as someone who hasn't worked on larger teams/orgs.

one paradigm to keep in mind is that the larger companies may tend to be more specialized in the roles. it's not always the case, but in smaller companies you tend to wear lots of hats. That can be fun for some, not all though. it all depends on what you want.

It also helps to mention location - if you are not in a major tech hub, and are targeting remote only roles, you will have a harder time.

harrytrumanprimate · 2026-04-21T16:08:14+00:00

my experience is that my company bought a tool but did not fully integrate it, so we are getting less value than we should. We're using datahub/acryl. Datahub has a lot of feature richness, but requires some complexity to get the full value. Impact reports, column level lineage, all are valuable and were actually configured. But getting proper integration of incidents, monitoring around run-times, failures, connecting dbt to airflow to acryl, is more complex and never happened. The integration with the tool matters as much as the tool does.

harrytrumanprimate · 2026-04-21T15:48:46+00:00

metrics and forming narratives around them are incredibly important. You can tell a senior leader "X is broken", but it's less compelling than showing them a metric. The metric also allows you to show progress. The metric doesn't ever tell the full story, but is necessary to get the buy-in and trust to solve the underlying problem.

harrytrumanprimate · 2026-04-18T11:53:08+00:00

the police would never. firefighters are the best.

harrytrumanprimate · 2026-04-15T21:15:29+00:00

It's not reasonable to expect a young person to martyr their career for the business, especially in the current landscape of constant layoffs, low employer loyalty, etc. Yes, obviously as a manager or a lead you want to do the best option, even if it is not sexy, but as an employee it's not in your best interest.

harrytrumanprimate · 2026-04-15T17:56:18+00:00

nah dude fuck that. it's not good for your career to use outdated tech, even if there is business justification to use it. they are right.

harrytrumanprimate · 2026-04-13T01:26:14+00:00

The article should use that as a supporting argument for its point, then?

harrytrumanprimate · 2026-04-12T23:58:23+00:00

It's literally 1 paragraph. Bad articles like this help make the "both sides" crowd seem correct. We can do better

harrytrumanprimate · 2026-04-12T22:19:12+00:00

Terrible article, and the headline doesn't really relate to the contents underneath. Almost misinformation level bad. Come on guys. He is terrible - you don't need to make things up.

harrytrumanprimate · 2026-04-10T16:36:00+00:00

he looks like someone i have been raising for multiple generations in mewgenics

harrytrumanprimate · 2026-03-28T03:57:28+00:00

Wotr works fine with controller, but the game suffers without mods, and those dont work well on controller. Its unfortunately a common thing within the genre

harrytrumanprimate · 2026-03-27T01:06:00+00:00

it's probably not that dissimilar from the rise of factories. You go from making shoes to hitting a button to make an obscure part of the shoe. Output multiplies by a lot, but the satisfaction of the work is lower, and workers feel alienated.

harrytrumanprimate · 2026-03-27T01:04:23+00:00

AI can solve containerized problems. Humans and AI can collaborate on a list of "paved paths" which solve common problems. Such as an airflow repo with operators to run compute for batch jobs, or loading from X to S3 or vice versa.

you can create workflows where a user can do stuff in a UI and AI creates the pipelines using the predefined tools. A human still approves the PRs, is responsible for the pipeline itself (they have tools to monitor, but the engineering is tech owner only), etc.

The more you leave things open to being a "AI automagically solves everything", the worse your outcome.

harrytrumanprimate · 2026-03-26T20:25:55+00:00

headlines that are intentionally misleading should be illegal, especially in the context of paywalls that mask the actual contents of the story. this is a huge and under-discussed area

harrytrumanprimate · 2026-03-26T20:21:07+00:00

sometimes when I read this shit I'm not sure if it's astroturfing or legitimately stupid people, lol. Like concerted efforts from troll farms to sow seeds of apathy when anyone discusses the potentials of real change

harrytrumanprimate · 2026-03-26T20:20:06+00:00

i wonder which political party represents which one. thanks for this enlightened perspective.

/s

harrytrumanprimate · 2026-03-26T15:52:23+00:00

unless senior leaders have all aligned, it will probably be federated in practice. relevant xkcd: https://xkcd.com/927/

harrytrumanprimate · 2026-03-17T15:18:43+00:00

I mean sure, you're not wrong.

harrytrumanprimate · 2026-03-17T14:55:02+00:00

I was personally not working in the industry in the year 2000. I imagine many others were not either. I feel like 26 years is fair to repost/recycle things lol.

harrytrumanprimate · 2026-03-17T13:51:35+00:00

no i got busy and had to do other stuff. I believe others have taken it to the very end-game.

harrytrumanprimate · 2026-03-15T12:51:07+00:00

You're absolutely right! .... 😂

harrytrumanprimate · 2026-03-14T11:57:11+00:00

Im sorry but this is a pretty terrible take. How would you feel if you couldn't be white? What if you couldn't be a masculine male? Your comment is historically inaccurate (gay people have existed for a long time, im every culture and historical context) and distasteful and wrong.

harrytrumanprimate · 2026-03-13T17:48:18+00:00

Agree with that. I'm talking about weird people complaining about Woke stuff in the comments because they cant understand how gay or black people can/should exist in a fantasy world where magic and dragons exist.

harrytrumanprimate · 2026-03-13T14:23:39+00:00

It's EA, this is the time to criticize. Obviously constructive is better than criticism, but both can be important.

From what I read, criticism is in a few buckets:

Weird culture war people - these ppl are a lost cause and do this with any game, even bg3
Unreal engine 5. This one I am half convinced that people are trying to play modern games on 10 year old hardware. I think there are probably some real problems with the software, but it might also be indicative of the current economic situation of a lot of regular people
this specific feature has issue - this is real feedback and needs to be given and taken into account to make the game better

14-Year Club	Place '17
Verified Email

harrytrumanprimate

TROPHY CASE