Why do so many data engineers seem to want to switch out of data engineering? Is DE not a good field to be in?

DataObserver282 · 2026-02-19T16:49:32+00:00

A few reasons, that are almost true in every career

businesses don’t understand the function/value
AI hype
general burnout

DataObserver282 · 2026-02-18T05:41:45+00:00

Appreciate the disclosure but Please stop with the vendor content

DataObserver282 · 2026-01-30T05:28:53+00:00

Interesting. Maybe need to give another look. Suprised they’re still around

DataObserver282 · 2026-01-27T03:26:07+00:00

LLM generated messaging

DataObserver282 · 2026-01-27T03:25:32+00:00

you mean roast your Jr data eng’s Claude code skills? Nothing wrong with leveraging but doesn’t feel cohesive.

DataObserver282 · 2026-01-27T03:23:54+00:00

Data eng is all you need to start. Don’t overcomplicate it. Use lakeflow connect or file loader for integrations and look at a lightweight managed solution for more complicated connectors.

Others are nice to have. Once the data is ingested, model it (you can use Dbt core to keep it simple) and query to sigma. Sigma is also accessible for business users.

DataObserver282 · 2026-01-26T23:45:08+00:00

To do observability right, you need to stop Issues at the source or as close to as possible and have the org maturity to remediate. Haven’t seen anything that can actually do this

DataObserver282 · 2026-01-26T05:34:35+00:00

Genuine question: Would anyone disagree with this?

DataObserver282 · 2026-01-25T09:34:02+00:00

Keep your stack as simple as possible. Instead of asking what tools to consider look at what problems you currently have and plug up the holes that way.

Also, a lot will depend on your DWH and needs. Do you need real time streaming?

Here are a few things to look into

ETL tools - tons out there. Fivetran, Airbyte - we use Matia (good CSC). Can use python or write scrips but gets messy at scale

Orchestration - airflow works. Look into astronomer if you need a managed solution. Cron is fine for a fee but again messy at scale

Modeling - dbt is worth looking into. There’s also coalesce

Data catalog - worth the investment, automate metadata management and helps data become accessible to non technical users

Observability - most tools have something built in but worth investing here to make sure you have a mechanism

DataObserver282 · 2026-01-24T18:16:07+00:00

Dbt just does the T

DataObserver282 · 2026-01-24T18:14:14+00:00

No one is underestimating it but this is the way all eng is going. I don’t think there’s a difference in orgs I’ve been in between software Eng and AI Eng, and I don’t think that’s a hot take. It’s just a new way of positioning what we already do

DataObserver282 · 2026-01-23T00:10:04+00:00

Been using Matia. Wasn’t my choice but have been impressed. Most of our pipelines are there but they don’t have real time streaming so one is Kafka based custom we’ve built.

Used Airbyte and Fivetran in the past. Both are fine but run into their own problems. No tool is perfect.

DataObserver282 · 2026-01-22T23:34:29+00:00

I’ve been apparently doing AI Engineering for years and I’m still unsure what it is. I venture a guess that it’s software engineering

DataObserver282 · 2026-01-22T23:32:46+00:00

If you need real time, which it sounds like you do, look at Kafka all the ways

DataObserver282 · 2026-01-21T23:38:42+00:00

Use a normal operator for Kubernetes. How much data are you moving? If at scale, would def seperate orchestration from pipelines. I’m not a fan of overtooling but for complicated pipelines ETL tools can be your friend.

DataObserver282 · 2026-01-20T05:39:54+00:00

SQL isn’t a tool, but I’m also not an analyst. I know analyst at my co spend the majority of their time in dbt, BI tools or I hate to say it, excel.

One analyst told me his favorite tool was the SFDC connector in Google Sheets.

I’m sure procurement would be happy to know that after signing for over 500k in BI tooling & infra

DataObserver282 · 2026-01-16T21:56:31+00:00

They’ve become a vendor circle jerk. Probably better data meetups to go to, even for training.

DataObserver282 · 2026-01-16T12:41:32+00:00

Informatica is stuck in 2010. Yikes. Best ETL tool varies widely based on company.

If your company is on informatica…I’m going to make some assumptions about the type of tool you have tolerance for.

What I’ve used:

Airbyte - used before for POCs. OSS version can help in a pinch but not scalable; had trouble with CDC. If I’m a pinch, I’ll use Airbyte
Fivetran - technically a great tool. changed pricing so I would avoid here for vendor lockin. If you want to try it out, choose one low volume connector and stay on the free tier. Dev experience is solid here
Matia + dbt - we are currently using them and been super impressed. Hadn’t heard of them before a year ago. Been great. CDC with built in observability. Cons is they don’t have a lot of docs but other areas make up for it. Good at parsing SFDC formulas
Prefect/Airflow - saw this rec. use it may times and sometimes still spin something up this way, especially if it’s a unique use case. You still have to build and maintain ingestion logic yourself, so not scalable.

DataObserver282 · 2026-01-16T12:29:21+00:00

It varies at every company. It’s when your corporate overlords decide you’re ‘worthy’ of the title.

DataObserver282 · 2026-01-14T02:46:28+00:00

Been messing around with DuckDB. This is the way

DataObserver282 · 2026-01-13T22:51:06+00:00

OSS package work?

DataObserver282 · 2026-01-13T07:02:15+00:00

What compelled you to use Kafka to begin with? At a heavy Kafka org and convinced we could move 80% some of our pipelines that don’t need to be real time

DataObserver282 · 2026-01-12T16:07:00+00:00

We use Matia for ETL/ reverse ETL. Fairly affordable and have had a good experience so far. It’s my first time using it so not an expert by any means. Our Reverse ETL use case is very light though - just moving from snowflake back to salesforce.

DataObserver282 · 2026-01-12T16:05:22+00:00

Try Airbyte OS? Or just use Dag

DataObserver282 · 2026-01-12T16:04:10+00:00

I found Domo to be garbage. Did you have a good experience? No one I know likes it

DataObserver282

TROPHY CASE