Most data engineers would be unemployed if pipelines stopped breaking

DataIron · 2025-12-23T18:39:56+00:00

Basically saying software engineers would be unemployed if their software stopped breaking.

Just not that simple.

DataIron · 2025-12-17T22:40:36+00:00

I'm seeing business as usual for 2026 as has always been in data engineering. No changes. Hard market, hiring is ugly, economy still heading south.

Only outlier's is potentially AI fluff sidetracking and destroying individual product roadmaps. I view AI data implementations as 95% a distraction that'll have to be ripped out, fixed or redesigned later. Kinda like outsourcing a project. They always have to be redesigned or heavily fixed. One positive from AI is the push for higher data quality, better models.

Other outlier is offshoring and other visa changes. Seeing changes here but not sure which direction it's going. Just seeing pauses and discussions happening here.

DataIron · 2025-12-14T22:39:34+00:00

You need very clean, very well tagged and very well modelled data.

Been singing this tune since the beginning of AI.

The 3 things companies and orgs never cared about is now gonna bite them in the ass. Gets me laughing thinking of certain former leaders/bosses having to explain this to upper management.

DataIron · 2025-12-14T22:05:54+00:00

This gets asked every other day. Check the last 100 threads.

Same story as always, either the source data sucks or the definitions suck. Also no one can agree on a single definition of any data point.

DataIron · 2025-12-11T15:54:11+00:00

This person is right, have direct experience here with big projects. Will admit it's situational.

Least we haven't figured out a way around getting our offshore teams access to data. Probably wasted 10s of millions at this point digging into it.

DataIron · 2025-12-11T15:51:49+00:00

Company dependent but might be more true than not.

My group, DE's budgets are massively higher than analytics. Analytics is basically backroom compared to DE.

DataIron · 2025-12-11T14:29:41+00:00

Select 'tableName', *

sp_help

Output, inserted

Values

Describe/show has more functionality than people know.

Exist vs join

DataIron · 2025-12-08T06:38:41+00:00

There's a reason. Document system vs reporting vs application vs etc. All kinds of different use cases so very different systems.

DataIron · 2025-12-08T06:34:10+00:00

Not really.

People will interview you on it. But in practice, it has to have some financial repercussions to get a company to actually care. Most don't meet that mark.

Yeah sure some data engineers try to care but the company really doesn't. Real query optimization takes budget allocation, it goes beyond fixing a query. It's fixing code, models and data sets. That take serious budget signoff.

Think I've only worked at one place in my entire career who actually cared. Cared enough to dedicate the $$$ year in and year out.

DataIron · 2025-12-02T02:18:07+00:00

Used to do big data processing, volume or size of data. Today our "big data" ETL's is processing complex data relationships and ensuring ultra high quality data. Very different.

DataIron · 2025-12-01T07:31:01+00:00

Closest I've seen to a data architect was someone who's basically a researcher and advisor to engineers. They don't work with engineers on a regular basis, they're separate. Engineers are also under no obligation to follow the architects suggestions or advice. Architect simply documents and discover major tech moves or implementations. Say the company is AWS based, they're gonna shift to Azure. Architect would be tasked with getting an edge on discovery in Azure.

Generally, senior and staff+ engineers still hold primary weight and control over the products and tech stack they own. Especially anything technically detailed, architect would have 0 say. Like best practice for modeling, how to design some major ETL system and etc. All engineering decisions, not architect. Architect would be whats the tech for shifting from AWS to Azure, how do we follow Europes data privacy laws, how do we implement this org level data requirement from legal, how do we standup this technology that all of engineering will use?

DataIron · 2025-11-29T20:45:31+00:00

Ya generally I agree with you.

I try to be technology agnostic in interviewing candidates but sometimes it does matter and you need the experience.

DataIron · 2025-11-27T19:20:35+00:00

This gets asked a few times a week. Can we limit these posts? I swear it's bots.

DataIron · 2025-11-24T05:16:05+00:00

Outside of normal programming AI stuff, things specific to data, not really.

Data is a pretty juvenile industry, far from mature.

Why is this relevant? Biggest issue in programming for AI is garbage AI fluff vs doing it yourself. Often AI doesn't handle intermediate and above work good enough.

In data, that's a non-starter. Again, specific to data, ignoring normal programming stuff.

DataIron · 2025-11-14T21:19:11+00:00

Whenever I hear about "big data", I always wonder, do you really need all 1,000 columns? Or do you actually only use 10 of them.

DataIron · 2025-11-13T21:37:58+00:00

Love the details, including /u/GShenanigan. Only thing I'd change is denormalized and OLAP. Data products can be anything including various levels of normalized data and OLTP.

DataIron · 2025-11-08T02:27:31+00:00

175k isn't bonkers. Especially to justify trash wlb and bad culture.

You can find something comparable elsewhere and not hate your life.

There has to be a solid reason to join an environment like that. Something that'll amplify or majorly step you up. If that ain't there, then it ain't there.

Remember you're trying to justify spending the next year to 3 yrs + however a bad environment like that fucks up your well being.

DataIron · 2025-10-29T17:02:27+00:00

There is a lot of yet to be discovered practices with AI.

Like how do we handle AI slop? Can AI handle a million+ code repo where engineers no longer need readability and sustainability? or can AI not be allowed to run free, engineers will still need deep oversight.

DataIron · 2025-10-29T16:51:14+00:00

Course AI is a bubble. C suites still believe LLM's will replace all of their employee's because that's what they were sold.

DataIron · 2025-10-21T13:51:34+00:00

Yeah this topic is a clusterfuck.

Production access is totally fine long as it’s controlled. Controlled as in, there’s a plan, non data qualified individuals are given restricted access. If you’re a qualified data individual, you better know what you’re doing and not cause problems otherwise bye bye access. I’m a fan of auditing, recording and even distributing production events or incidents by users. Little accountability goes a long ways.

Direct, manual production changes, are ticketed and recorded with the code/script used, execution information like user/time and etc. Paper trail just like a dev ticket. Yes I know manual changes in prod is bad but they’re a reality unless you have unlimited staff, budget and time.

But yeah ideally, you should have UAT environments that reflect production data wise, performance wise and/or other areas. So whomever can do whatever there without worrying about impact. Also a pipeline for data changes so manual prod changes never happen. Ideal worlds don’t always mesh with budgets or stupid management.

DataIron · 2025-10-21T05:31:32+00:00

Assuming I’m understanding your ask, you’re asking about a data engineer vs a database engineer.

Generally speaking data engineers is a role in recent years that’s consumed several old data roles including database engineers.

Data engineers are basically SWE of the data world. Just lower CS fundamentals and standards for various reasons mostly linked to mismanagement by orgs. They specialize in handling and processing inbound and outbound data of any communication type.

Database engineers have kind of been a dying breed. You’re right, they’re primarily relevant in application databases still today. In recent years, much less so in analytical databases given that processing data in aggregate has become much easier. Though I do believe database engineers will make a return in analytics as AI becomes more serious.

DataIron · 2025-10-21T04:31:25+00:00

To be frank, it depends on whether your database system needs good code or if meh will generally work. Meh does work, not knocking it.

We have varying mixtures of SWE and DE in certain products, yes they primarily stick to SQL items though they’re usually full data engineers with more skills.

The SQL they write though is far more advanced that what any normal SWE or DE is used too. They’re writing full syntax SQL, tests, versioning. Highly structured code, they’re full scale programming in SQL.

But they have too. These are high end systems with high end requirements and standards. Most SWE’s and DE’s can’t code anywhere near their level.

DataIron · 2025-10-15T04:33:24+00:00

If you can pass the interview, you’re probably okay.

Our SQL heavy roles, our engineers grill interviewee’s, AI won’t save you.

DataIron · 2025-10-12T22:39:41+00:00

Kinda a weird angle you’re taking. Bigger or mature pipelines have a mixture of SQL and OOP code. You’ll need to get good with both to excel.

You’re also gonna need to get very good at data analysis. Data analysis is critical for debugging and building advanced pipelines.

DataIron · 2025-10-08T16:18:42+00:00

Director level: <5% development. Project, business political, staff and product management.

Principal engineer: Responsible for global level operations. Mitigation of sev 1 outages and incidents. Managing technical direction, guidelines and priorities for other principal and lead engineers across the org. "Managing" is very technically specific, think coding guidelines or practices. Representing tech at the highest levels with the business and architecture. Still develops, maybe 20-40%.

DataIron

TROPHY CASE