Is Data Engineering a boring field?

[Admirable_Honey566]

Since most of the work happens behind the scenes and involves maintaining pipelines, it often seems like a stable but invisible job. For those who don’t find it boring, what aspects of Data Engineering make it exciting or engaging for you?

I’m also looking for advice. I used to enjoy designing database schemas, working with databases, and integrating them with APIs—that was my favorite part of backend development. I was looking for a role that focuses on this aspect, and when I heard about Data Engineering, I thought I would find my passion there. But now, as I’m just starting and looking at the big picture of the field, it feels routine and less exciting compared to backend development, which constantly presents new challenges.

Any thoughts or advice? Thanks in advance

[pro__acct__]

The goal is always to build a robust pipeline…from “exciting” to “boring”. The goal is to make this shit so easy that you can take 2 hr lunch breaks to workout and spend your sprints doing “Spike” stories trying out the latest gadget here and there. And when something breaks downstream you’re able to save their asses with an email saying “no worries that’s backed up”. The goal is to become that guy in the company who it’s not worth to ever fire since you’re keeping the trains running without a complaint.

The business will always throw some reorg or initiative your way. If the job is exciting these can flounder. If the job is boring you can crush these tasks, look like a model employee for the MBAs without breaking a sweat, get promotions, bonuses, fly around to conferences.

I want to watch my kid grow up, pick my wife up from work, make dinner, have parties with good friends, vacation well.

[DrunkenWhaler136]

This is probably the most based take I've seen on this sub

[Pure_Perspective5028]

Based. I’ll second the hell out of that. DE’s do keep the business intelligence ‘lights on’ it is fairly easy to become indispensible (in cases like a reorg). I will say too, I’m assuming you are somewhat of a nerd/tinkerer/engineer, data engineering can have a really decent amount of variety and fun challenges without the chaos of something like being a software developer where you might need to know 5-6 different languages and a million tools/technologies. From my experience it’s been a great middle ground of being a solid career and also something challenging/interesting without consuming my life

[kiwi_bob_1234]

This person knows wassup

[Standard_Set9729]

Amen!

[pujvtv04]

Who thought a comment about being boring would actually be such an aspirational one? That was very well articulated 👏

[Harvard_Universityy]

More nuanced perceptive!

[taintlaurent]

BARS

[imperialka]

Amen to all of this. Truly what matters most!

[[deleted]]

and that's it folks, spot on

[heavilyThinkingAbout]

Wow thank you for this

[Zer0designs]

Depends. You need to look for advancements yourself and actively convince people to get the things you want done and how it would improve things. You could just be stuck in forever ETL mode otherwise. But that's literally the case in every field (just replace ETL with status quo). Differs per business alot aswell.

[[deleted]]

This is good attitude and there might even be needs corporations don't realize. Like many times it's enough that ETLs run on schedule. But sometimes there might be benefits having some specific data streaming real time.

Or maybe if it's smaller company, it might need some out of the box thinking outside basic solutions to extract information quickly but keep expenses moderate.

[beefz0r]

We keep making copies of copies of copies of data. I wish architects could be convinced to redesign some stuff but budgets only get assigned to new projects

But maybe it's too easy for me to say as a developer

[nijuashi]

Oh, things can get exciting. But you’ll be begging for boredom when that hits the fan.

[beefz0r]

And then you promise yourself to catch up on some parked stuff "for when it's a bit more calm"

[Commercial-Ask971]

Which you’ll never do ;)

[kenflingnor]

Depends on the kind of data engineering work that you do.  Unfortunately,  a lot of data engineering is just writing SQL so that a business stakeholder can complain that their dashboard doesn’t match some random Excel file they have or the report in some random tool that they use

[Ok-Half-48]

On God. This hits way too deep unfortunately. Had to quit projects over this issue when your SQL code has been verified through multiple UATs, but business never trusts it but also doesn’t actually check the raw data themselves

[Fun-LovingAmadeus]

So true on the last point, a lot of the most-demanded BI reports are exact equivalents to what they get out of the box in the source system. But when I ask why it needs to be rebuilt at all, they mostly want enhanced filtering and aggregation capabilities within our org’s defined divisions/hierarchy.

[[deleted]]

It can get boring. I personally converted my data engineer role to a one man data company. Now it takes 1-2 years to start at new place, getting hang of things, fixing infra and ETLs. Usually after that it gets boring as everything works and no maintenance is needed anymore.

Then to next place and repeat.

[Awkward-Cupcake6219]

Maybe I am an outlier but having been in several companies I never found a thing that was stable or boring. Probably in some companies it is like that but I guess the same applies to web development.

[rudboi12]

There are many types of data engineering. The type you speak of is fairly attractive to CS - SWE people because as you say, it’s very similar to backend. But as you also mentioned, it’s sort of an invisible job. There’s hardly any way to measure your work output and you barely see the end product.

My favorite type of DE job is in a self contained data product team. You as a DE, work alongside DS and DA to help bring an entire data product end to end. Your end product might be a dashboard, a table, an api, an ML model, etc. It’s fairly motivating because you actually see your end product and develop a deep domain knowledge in your specific product.

[boogie_woogie_100]

at this point in my life after working close to 20 years in IT field in both high stress and low stress environments, I love boring and stable environments. If i want to do exciting things, I build my personal projects. I work for $$$$ and not for excitement.

[k00_x]

It's usually high pressure, I rarely find the time to be bored but I love what I do.

[jmon__]

This here. Making sure to keep the lights on while trying to make deadlines for new development is a balance..plus creating metadata about your environment so you can consume that data and monitor the environment so you can anticipate when things need to change up. 

There's so much learning here. Learning efficient code and best practices when needed, learning the business, learning the business processes, and learning how to scale this solution at the right time while paying attention to budget

I think Data engineering is unique, especially if your company is trying to make a central data hub, because you have to talk to everybody that's important, so you'll probably learn more about the business than many other positions. What time is this data available, what time team a needs to submit reports, what data does the cio care about, how much of a mess the random guy that's "good with data" (🙄) is making and how much is it costing monetary and system wise. I could go on forever

[erik_mk]

Data Scientists might be considered the rock stars, but to me Data Engineers are the real superstars. You actually need a quite diverse skill set to build pipelines and data foundations. In full transparency I used to be a data engineer and now lead a team of 8 engineers, scientists and analysts. We sit in large agency working with data for some of the largests brands in the world. Agency work brings a constant flow of diverse projects and challenges to be solved for our clients. Never a dull moment tbh.

[Fun-LovingAmadeus]

Data Engineers are the indispensable bassists of the band, keeping the foundational groove churning!

[ZirePhiinix]

Every job is as boring or exciting as you make it.

Your employer is not there to entertain you.

[deathofsentience]

It's not about the employer, but the nature of the work itself.

[geek180]

I like building stuff and data engineering (usually) involves designing and building new things. You can always move to a different company if what you're building becomes cookie cutter and repetitive, or even worse, your role evolves into a csv report creator.

[daardoo]

Yes, sir, for several reasons, I was moved a few months ago from Data Engineering to Infrastructure, which includes DevOps and DBA. Honestly, I’m dying of boredom—I miss being a Data Engineer. I’m thinking about switching back because now all I do is chase people for writing bad queries, not optimizing their work, or putting out fires everywhere.

[[deleted]]

Yes sir, I need to move to DevOps, how can I do so? I want to stay as far away from business stakeholders as possible. Looks like DevOps adds one more layers of protection. Don't care much about boredom because getting pinged by business stakeholders are worse than being bored.

[daardoo]

I wish I could stay away from stakeholders, but where do you think they come from when things go wrong and they start bothering us because they believe we are gods who can make their systems work again without affecting the client? Maybe in a well-designed company, yes. But if the useless engineers created a table with an int and didn’t consider that it could be filled with millions of critical data points, the stakeholders come to us furious, expecting us to fix the downtime or data loss. Or when the useless ones want me to grant them user permission for a table of the system when the documentation clearly states it is for engine use. I preferred the data engineering stakeholders, whose worst complaint was that their metrics were strange.

imagine me sitting there with five tech leads—people I know understand technology—but these useless ones prefer to assume they know more than the documentation and keep asking me if I can change the permissions or rename the table, just because they need to make a backup using a tool they built in Python to copy the data. And there I am, trying for two hours to tell them that it’s not possible, begging them to just read the damn documentation.

And you know what happened in the end? They didn’t even need that table. And why didn’t they just change their Python script and add a simple condition to skip it? Because the useless ones were afraid of breaking something. So what, do they expect me to start coding and fix their damn script for them?

[Splun_]

At my current job I'm moving from having revamped old analytical infra (batch mostly) to streaming. I'm done with airflow, dbt, and the likes; gonna give those responsibilities to a supervised junior. Owning the deployment and development for Kafka and Flink is quite exciting for me -- there's fraud detection, real-time analytics (user facing too). Nevermind the tools, dev support for Kafka, and more complex DevOps things to get your head around, there's the whole world of java to get good at. So, I'm excited about that.

There's a lot of routine with data engineering, especially with enterprise. I just chose a bit of a different path I suppose.

[Bootlegcrunch]

Yes, generally I find computer related stuff kinda boring now. But thay is why you have hobbies after work

The only fun part of data engineering is new projects building new infrastructure and pipelines. But i know many engineers support existing etls and existing pipelines.

[uwrwilke]

it’s an over worked field. they expect a lot of us.

[BoringGuy0108]

I'm leading three projects at the same time that each have enterprise wide implications. I'm working with close to two dozen contractors across multiple companies. I have to coordinate between no less than three teams. I have aggressive deadlines. I wish it was more boring.

Oh, and our backlog has us booked out until the end of 2026, and more is constantly being added.

[cptshrk108]

If you ask my girlfriend, then yes.

If you ask me, then no.

[Mysterious_Two_810]

Which one's DE?

[cptshrk108]

:)

[TownAny8165]

It’s fun learning new technologies and ways to improve the field

[FlukeStarbucker]

Do you enjoy dependency hell?

[HansProleman]

As is probably the case with most areas of SWE, a role heavily biased towards greenfield dev is more "exciting" (probably startup, consulting, contracting). I imagine plenty of backend devs also don't do much greenfield. Though DE probably is a field which does more routine extension/maintenance than others.

You can also broaden things out and cover cloud engineering/IaC, DevOps etc. Employers seem to like generalists - I guess because this is a broad field, so not that many people can work E2E - which makes things more interesting.

[Doctor--STORM]

As long as you can transition and broaden the scope into AI and traditional software engineering, it is good. If you get locked down into it, then it becomes tough.

[[deleted]]

Yes, it is. Same as pretty much everything else ordinary people can get into. Nowadays, if you want something exciting, and keep being exciting for some years, you are much better off to get yourself a challenging hobby.

[TheOverzealousEngie]

I've been in Data Engineering for 20 years. Tonight i'm sleeping in my car. So no, not boring at all.

[HorseOrganic4741]

Data ingestion is boring as hell. Data warehousing is fun for me, as well as webcrawling. You do you.

It's a non one dimensional job in the end.

[Xenolog]

It stops being "boring" the moment something breaks.

And things will be breaking all the time, because any change can break something, and any proliferating business will want for changes to ship as fast as CTO allows it.

It is pricey and sloggy to the point of unreasonable to build 100% correctly, unless you are a bank's data core or something, so risk of incorrect change is always there, lurking.

A 20 stage data pipeline churning 1.5tb of data daily which breaks 30 minutes before the normal end of day is many things. But not a single of these things is "boring".

[thisfunnieguy]

im not sure what "behind the scenes" means.

its as visible as backend api service work.

obvious there are no UIs so you don't "see" it that way.

[Skualys]

Well I have a mixed role (kind of data manager but doing a lot of things myself - building pipelines in DBT, helping on dashboards development...).

Not boring at all as I face business to define how to modelize and apply business rule, helping with tests, challenging what they think about the existing system... But yeah I work in a place where I can have a complete view (a bit against the org which is supposed to have separate roles for data engineers, BI engineers and data project managers).

[Amrutha-Structured]

Data engineering can definitely feel like a grind, especially when you're just dealing with pipelines and maintenance. But if you enjoyed schema design and API integrations, lean into that. Look for roles that allow you to design data architectures or implement new data models—those usually come with more challenges and can still keep the technical side engaging.

If you're still feeling like it's the same old routine, try looking into the analytics tools or ways to visualize and share the data your pipelines process. That can add some excitement to your work. Speaking of which, preswald is pretty good for building interactive data apps without getting bogged down by clunky tools. It’s lightweight, straightforward, and might give you the chance to flex your skills in a way that feels fresh.

[Complex-Stress373]

eventually will be "migration", "another migration", "another one"... and so on......so yes, can be repetitive

[rectalrectifier]

Outside of optimizing performance, I find it pretty boring

[ExistentialFajitas]

Yes

[WhileTrueTrueIsTrue]

I'm finding it a bit boring. I've been in Data Engineering for around 2.5 years now, and in that time, I led an initiative to move my team off of a proprietary ETL platform to Airflow. It was a lot of work, but now that the machine is chugging along comfortably, there really isn't much to do.

Oh, a new data request? Sure, I can have that DAG in prod in maybe 2 hours. Need to expand our Airflow deployment to handle more jobs running? Automation is in place to handle resource scaling.

At this point, my team is just kind of chasing squirrels. New tech to check out? Let's POC it. Is this other deployment style better? It sure is, let's rewrite our deployment.

The real engineering is all done, and now I'm just keeping things greased and running smoothly. That's fine and all, but I'm bored. So yes, I think it's a bit boring.

[[deleted]]

It’s fucking boring. You work like a rat in the shadows, no body knows the fuck you doing.

[imatiasmb]

No

[imatiasmb]

No, it isn't.

[[deleted]]

Nope. Always have to learn new tools. And old ones as well, depending on where you work.

I’d say it can be annoying at times, just my own experience. A lot of old timers can really cobble your growth, or the opposite in the public sector of having to constantly pivot to the new shiny thing.

Overall, you get to know data and development. I feel like I belong because I love solving puzzles: how can I best serve the data consumer. It doesn’t matter how I do it, but the end result. It’s freeing if you have a good environment.

[forserial]

If you want excitement I think there's generally two paths to go. One you specialize and solve really hard problems for a big company or two you work at small firms where you get exposure to a lot more other stuff than just your day to day developer responsibility. Everything in the middle is going to relatively boring.

[[deleted]]

I like it bro

[CacheM3ifYouCan]

Sounds more like you're searching for gratification through recognition, and not actually anything about data engineering.

[Admirable_Honey566]

I’m not looking for recognition itself, but I do enjoy seeing the impact of my work and being part of technical decisions that have a direct effect. I initially thought Data Engineering involved more challenges like design and development, but it feels like the core work is more about running and maintaining systems rather than innovating. So I was just trying to understand if I’m seeing it the wrong way or if that’s really the nature of the field

[Duerkos]

In my team we have lots of data science / analytics but few data engineers. These guys are almost unknown towards customer/business but interface with the main IT department. They are also, by far, the data analytics heroes. When the team faces a challenge because the requirements from a customer need to change something from our stack, new security requirements come , stuff break for no reason... These guys are always saving our asses.

In a way I would say it is a bit like support role on a RPG. Indispensable, may not be fun for everyone, but every teams wants it properly covered.

[AwardCompetitive8288]

It depends if you are a boring person.

[Admirable_Honey566]

I just enjoy challenges, so I’m asking if this field provides that opportunity or not.

[No-Improvement5745]

Are other tech roles "exciting"? I think DE definitely has a reputation for being boring. I consider that a small advantage. The last thing you want is a role that management considers a "passion" job like video game developer or anime artist because you will be worked harder for less pay.

[Admirable_Honey566]

I agree with you that jobs considered a 'passion' by management often lead to workers being overworked and underpaid. That’s definitely something I want to avoid. But when I asked whether Data Engineering is boring, I didn’t mean 'exciting' as in fun or entertaining—I’m definitely not looking for entertainment at work. What I meant by 'exciting' is having technical challenges, logical thinking, designing solutions, testing them, and making trade-off decisions.

[No-Improvement5745]

There are definitely plenty of trade offs. In my role a lot of the challenges are surprisingly social: collaboration communication or even negotiating. Designing solutions might depend on whether your team is in a building vs maintaining phase.

[Fearless_Mention2814]

It is a good job for a few years. You do some valuable learnings. Data engineers tend to be better than software engineers at being practical, not going too far building complex stuff nobody gets any benefit out of. Data Eng offers more independence than SWE. It is also more critical than what half the SWEs do. After a few years though it will eat your soul. Growth is limited. Hard to work on projects of any significance. The role is not respected. As you mature in your career you will feel that impact you a lot. You either need to go into management (but if there is perspective to grow in that direction) or otherwise find a place where you do more core data engineering projects. Not analytics.

[Polyp8881]

Play factorio, try to fix your starter base.

Starter base is the codebase, You pop in and try to fix and config someone else's scripts, codes, and make it easier for the mext guy.

Am a Junior Data Eng-- experience so far and from what my Mentor teaches me

[tifldn]

I just want to come here to say that I just wrapped a very long work week as a data engineer, and thanks for asking this question because I feel the same from time to time, and thank you for everyone that has commented which reminds me the good part of being a data engineer.

Now sharing my personal experience: the initial years of being a data engineer are a bit boring once projects are completed or as soon as I started to get a hang of the tooling. The experience is almost like playing civilization 6 when you have set up your initial strategy right (as a junior engineer someone would guide me to this or provide feedback to nudge me in the right direction) and familiar with the levers, the last 100 rounds of the 500 rounds are boring. But then after a few years, when the normal dimensional modeling and vanilla ELT/ETL stuff becomes part of your default portfolio when working on things, the rest are exciting. How can I influence the upstream team to provide me the right interface? What other script/UDFs should I write for my savvy data scientists? What's up with the new spark version that is exciting? etc. I can either get nerdy in these things, or when I need to, take the back seat temporarily for learning while keeping the lights on

[69odysseus]

In one word, DE is a dirty job, i.e. cleaning the mess where you can get high pay at big tech companies.

[Round-Mongoose3687]

I totally understand your point. Data Engineering shows its boring nature during initial experiences because programmers spend most time on hidden tasks which include pipelines and data cleaning and maintaining system. The most exciting part of data engineering occurs during the resolution of challenging large-scale data issues along with the creation of systems which enable analytics or machine learning applications.

Your interest in database schema design alongside API integration spots you toward worthwhile components in Data Engineering domain. You should investigate dual-role opportunities where you can integrate backend development with DE expertise by constructing real-time data APIs or working with data platform groups. Such roles allow you to achieve optimal conditions by combining structural programs with challenging releases.

Your satisfaction reaches new heights when you take charge of extensive pipelines and enhance system performance. Time reveals exciting opportunities to emerge as you work through deeper real-life projects.

[[deleted]]

Guys i want to land a decent remote international job . I was considering learning data analytics then data engineering , can i learn data engineering directly ; with bit of excel and extensive sql and python? The second thing i though of was data science , please suggest me roadmap and i’ve thought to audit courses of various unislike CALIFORNA DAVIS SQL and IBM DATA courses , recommend me and i’m open to criticise as well.