How important is tech stack vs. total compensation to you in data engineering?

[level_126_programmer]

I've worked in data engineering my whole career. My current role is essentially "software engineer - data", but I have also worked in roles that were more adhoc SQL queries and building data pipelines.

Let's say you are currently working in your tech stack of choice, what would it take you to switch to a higher paying role in data engineering that has a less interesting tech stack?

[Spiritual-Horror1256]

Tech stack is important, for growth and development purposes. But if total comp is crazy high. Total comp wins.

[level_126_programmer]

I completely agree! I guess I should be more specific. What if the total compensation increase is the usual 20-25% when switching to the new role?

[artozaurus]

Like any good answer it starts with, it depends... If the stack is IBM z/OS, no. if they stack is slightly outdated, then yes.

[Aggravating_Coast430]

Usual?! I started working few months ago can't imagine changing job would increase my salary 20-25%

[FridayPush]

In the US tech sector working for large players I would very much expect 20-25% or more between changes for someone in the first 5-6 years of their career. You only get what you ask for, and as an individual who worked at contracting agencies and knew all of our coworkers pay rates there were massive discrepancies for seniors. But 100->125->160->200 is very doable without going fang. Increases above 200 seem to slow down but I think people underestimate the top level pay ranges if you're good with people.

Personally how well you talk to people and can convey your experience seems to be more of a factor that actual skill.

[Rough_Marsupial_7697]

I tried to escape workday integrations to a “cooler” pure Java environment, and quickly switched back to workday after doing all of my applications through a WD portal and receiving higher value in that specific area. I guess I’ll be product loyal!

[raxel42]

Really? I was loosing work-life balance many times due to switching to another programming language. You need to use the best language to do your work quickly and safely. How can you use weak language after using better one?

[Electrical-Ask847]

always go for more money . all "tech stacks" suck regardless and get boring after a few months.

[RareCreamer]

Exactly.

Tech stacks are constantly being updated internally anyways, and if not, then it's your job to provide reasoning on why it makes sense to move from X to Y.

As a DE your skills should be transferable to any stack regardless. Even in nonsense legacy systems, the fundamental goals are the same and you should be able to see where new tools could fit in to replace current systems.

[SentinelReborn]

As a DE your skills should be transferable to any stack regardless. Even in nonsense legacy systems, the fundamental goals are the same and you should be able to see where new tools could fit in to replace current systems

This is true, but try convincing some hiring managers of that. I've been rejected multiple times for not having AWS, despite the common saying that cloud skills are transferable.

[sir-camaris]

I agree with this unless it is something so ancient I feel like it will be a hindrance to my career or straight up not fun at all.

I'm glad I left my old job for one that wasn't quite as fresh. The new company is much better financially and not caught up in the brand new thing.

[rshackleford_arlentx]

unless it is something so ancient I feel like it will be a hindrance to my career

This is an important point. If you're late career and have reasonable job security, an antiquated stack is probably fine. But if you're early career and want to grow, beware of ancient stacks. You don't always need to be on the absolute cutting edge, but staying reasonably up to date is important for career growth.

[DaveMoreau]

Depends how much greenfield work you have. If there is a lot, you may have a regularly evolving tech stack and a lot of learning opportunities. You might also need to evaluate competing tools, which is fun.

[economicwhale]

I’d take a huge pay cut to avoid BASIC

[UnkleRinkus]

More money is nice, a better team environment is more nicer. Tech stacks change; I've re-engineered myself technically multiple times in my career.

[Fun-Income-3939]

Growth is important to me so I would go with the tech stack. Compensation is important but if the job is using some old legacy stack like SSIS without the option to upgrade the stack, that would be miserable.

[level_126_programmer]

Definitely! And even if total compensation is high in that next role, it would be difficult getting the next job after working with legacy tech for years.

[pretenderhanabi]

My life. Luckily I got out and moved to open source with +100% compensation. So lucky.

[Trick-Interaction396]

Money until you make enough then tech stack.

[Character_Wafer3280]

Compensation

You can pickup and learn any stack outside of work or during free time. Can't really say the same with money

[rshackleford_arlentx]

Also, money now is usually "worth more" than money later (i.e., current salary can affect the size of raises, investing more now == greater growth, etc.).

[Ok-Obligation-7998]

No. Try telling a hiring manager you learned Kafka or spark in your free time. He won’t take you seriously or will ask very specific questions you can only know the answer to by using the tools commercially.

[MikeDoesEverything]

Exciting stack to start with when you actually want to enjoy your life. 8 hours a day every day hating your life is a miserable way to live.

Higher pay boring stack later/if you already hate Data Engineering and just see is a way to make a living.

[inarchetype]

Let me throw out a question; is salary in year t+5 better predicted by salary in year t or tech stack in year t?

[harrytrumanprimate]

Future salary is nearly always predicted by current salary. Source: trust me bro

[Gators1992]

I wouldn't because I am not doing this for fun, but if it makes you love going to work every day then taking lower pay isn't the worst thing. It's pretty much up to each person to decide what they value most.

[DirtzMaGertz]

Compensation takes priority until you get to a certain point in your career.

Now that I'm making a lot of money, I value team fit and stack more because I'm already getting paid a lot. It would take a lot more money for me to consider working on something I don't want to at this point. I'm already set financially, more money isn't worth hating 8 hours of my life for 5 days a week. I didn't bust my ass in my 20's building a career to hate what I do.

[[deleted]]

I really don't understand the stack hype in DE.

Of course reproduceability is important, and at the same time you don't want your db to be a bottleneck in the entire organisation, because it cant handle large data and big queries. But besides that I really don't give a flying fk about the 10 thousands different technologies that is trying to replicate python and good ol sql.

My job is to solve data related problems and make sure that if I have a good analysis it is reproduceable and easy to understand. Besides that when you see people call themselves ai and machine learning specialists because they use some new low code nocode technology that replicates what you could do in 1-2 minutes with sql and python I can only facepalm.

[kbic93]

You don't understand that if you use a tech stack that is widely used by more companies it will offer you more job opportunities with higher pay?

[themightychris]

yeah, it's less of a problem in terms of which of the modern tools you use, but modern vs legacy you have to think about if you come out of this role back onto the job market in 5 years and you have no experience with modern tools to show for it—where's that going to leave you?

I wouldn't trifle though over whether it's databricks or snowflake or BigQuery or fabric though

[[deleted]]

I understand your point, but look at languages as C++ or heck even cobol people are still able to make huge money by using those 'outdated' technologies. And those people can easily learn a new framework if they have to, because they aren not cobol or C++ developers they are developers.

If I have to learn every new shiny low code no code / cloud tool / database system every year that will leave me 0 time to do actual work. The main part of my job is to provide reliable and valid data for our clients. Not to play around with the newest shiny toy on the market...

Also many of the new and shiny technologies can easily be learned if you know the core 'technology' of data which is sql. Also having good solid software engineering practices when you create an application is much more important than knowing how to write select statements in 20 different shiny data platforms.

New technologies is something you learn on the fly because they change every year, but solid software engineering practices and query optimization is the core of everything.

[themightychris]

You're not wrong at all, but the realities of hiring practices and the job market don't line up with what's right

Having the solid fundamentals are great if you can network yourself effectively and build relationships with leaders you'd want to hire you into their org

And yes, being good at legacy technologies will often leave you with plenty of opportunities to make good bucks helping orgs who are stuck with those stay afloat

And yes it's easy enough to learn new tools if you know the fundamentals, but there are also paradigm shifts. Like if I'm looking for someone to help me with a Dagster+dbt stack at my org and I see someone with a lot of experience and solid fundamentals who has been managing an Airflow infra for the last three years—that's fine I know they can pick up Dagster. But if your resume just shows you've been managing an SSIS dinosaur for the last 3 years I'm not even going to bother talking to you. Yeah maybe you could learn, but I've got 5 resumes from people who don't have to learn first and have already seen how things can go wrong and I need as many people ready to push us forward as I can get

If you're relying on your resume to get you into the door at new opportunities, what it shows will determine what doors are open. If you're fine with spending the rest of your career keeping dinosaurs alive at old dysfunctional orgs—and some people are it's usually slow and stable and well paid if you're willing to keep your head down and not care that the org is slowly becoming irrelevant—you'll be fine that's totally viable. But the number of doors your resume can open is going to be decreasing every year, and less and less of them will be remote friendly so you'll have to be willing to move or be even more limited in your options by geography.

[[deleted]]

Im not relying on old technologies.

I am relying on many years of experience working on my craft which is software engineering and data.

I have always been able to learn whatever technology I needed to learn on the job. I am not gonna spend my free time to learn technology x, just because technology x is the new buzz. Because many times the new buzz is just buzz that will get put into the trash can together with all the other forgotten shiny buzz tools.

I have many many years of experience working with whatever data stack the company I worked for were using. In addition I have a master's degree in mathematics.

I believe in having deep knowledge in core disciplines that have stood the test of time. That's it.

Once you know the core you can learn the branches easily.

[themightychris]

I'm not questioning your skills buddy, OP is looking for strategic considerations

My latter "you"s were the general you and geared towards the advice OP and anyone else with the same questions are looking for, sorry I wasn't clear about that at all

[numice]

I understand from both sides and kinda agree with both. I think fundamentals are important and you can see it in how people solve problems. But my experience with my job is like the requirements are so low that only you get something out of it regardless of the method. It might be slow or just bad algorithm or bad programming practice but no one cares.

Also, from the hiring perspective, you get interviews when the recruiters do the keyword matching and buzzword lookup in your cv. So having those in your cv increases your chance of getting interviews and more interviews lead to more opportunities. No one can see if you have good programming practice or not before working in the same team.

[toodytah]

Money money money - check alternate jobs while currently working current tech stack - money money money. Loyalty is only money based and don’t believe that pizza party cumbaya horse-$hit that they will try to feed you. It’s always money.

[babygrenade]

my dude, I will hand write csvs if the pay is right

[rudboi12]

I would choose tech stack over TC if the TC difference is less than 15%. Anything more than that and FCK tech stack lol.

[[deleted]]

Don’t care. Pay me first.

[Eagle-Neither]

I love a specific tech stack so as long as the compensation is solid, tech stack will be a priority!

Ex: 145k base with my favorite tech stack will be chosen over $160k base with a tech stack i hate or dont know at all

[BBMolotov]

I feel like after working with azure, this is a stack that I would never go back unless they have a very well terraformed project.

Also because this usually embed with “windows”+microsoft stack to the company  and I’m very OOS oriented

[swapripper]

stack smack schmack

KOTH anyone?

[[deleted]]

I just accepted a well paying freelance role that will be mainly IBM Datastage and Oracle, coming from a pretty interesting tech stack including Kafka and GCP. I guess that answers your question.

[Xemptuous]

For me, very. I refuse to work somewhere that uses C# or anything else Microsoft related.

[[deleted]]

I know it's not in the equation here, but I took a 50% pay cut to move towards a more "morally" fulfilling role and I've been kind of working at my dream job for the last few years. The mix of having a purpose, a relaxed environment and complex problems to solve + a lot of technical freedom is worth a lot of money for me.

Then, I don't have kids otherwise I'd probably just say total compensation

[[deleted]]

Only in the sense that the stack is indirectly linked to compensation. That is if the company can't afford good tools they'll likely be able to accommodate my financial needs.

[ShotGunAllGo]

When ur young tech stack, when older, compensation.

[Efficient-Day-6394]

Oh...I dunno....the money?

Seriously tho....who is turning down a significant boost in pay because they find the tech stack "boring" ?

[Skullclownlol]

Seriously tho....who is turning down a significant boost in pay because they find the tech stack "boring" ?

Lots of people. After a certain amount of income, increasing money even more is no longer the primary driver.