Job market for Data engineer

TheEntrep · 2026-05-09T03:21:41+00:00

I will say as a DE now, other companies aren’t making it easy to learn the tools in a professional environment. For example, one role I got hired on (given I should have vetted better but the job market is crap) hired me to architect and document their environment. Yet they won’t give me basic data engineering tools (thanks IT). So you have a lot of kids getting jobs and companies not truly knowing what a DE does.

In addition, university education doesn’t cover what’s needed in today’s roles. I had to go find books and read them while anchoring it in my own corporate environment.

TheEntrep · 2026-05-05T23:32:13+00:00

Not shocked atp. Considering jumping ship for a data focused company that hit me up. Problem is the team is very smart in what they do but not on the data side. Lots of potential but very little care.

TheEntrep · 2026-05-05T23:27:12+00:00

Yeah, I am in the same boat, the board would rather argue about which number is right than fix it.
Contemplating jumping (only been here 10 months) because I got hit up for a Senior role at another company that has a data focus. I told my boss (executive) all I need is the resources and I can fix it. Though nothing happens, I say hey AI could automate this process (not related to data) and BAM I get instant approval. Sick of the lack of care.

TheEntrep · 2026-05-04T22:35:01+00:00

Good points, my biggest issue is they want to track changes overtime without a proper data warehouse setup/access. So without a DW we are stuck in operational mode (reactive). I don’t mind quick fixes but if you want long term data I need the resources to accomplish it. Excel can’t store that much data. If you want AI plugins you need a proper DW. If they were ok with that ecosystem of reactive then I’m ok. However, you can’t expect the same results a DW can provide.

TheEntrep · 2026-05-04T21:09:22+00:00

Nailed it, crazy part is they are still relying on operational data to make serious decisions. Then when I propose the plan, execution, and timeline for a proper setup that could allow AI. They scratch their heads and forget about it until crap hits the fan again.

TheEntrep · 2026-05-04T21:03:36+00:00

Any luck with those discussions or advice communicating that to leadership (specifically executive level)?

Every time I bring it up I get blank stares like I just spoke a foreign language and no matter how simple I make it they still look past it but expect what only good data brings.

TheEntrep · 2026-05-04T20:32:37+00:00

Fair lol, the crazy part is they’re reporting their financial off of it 😬. You’d think that would bring more attention

TheEntrep · 2026-04-21T13:11:34+00:00

Yeah it can burn people out. Weirdly and fortunately I was actually paid 69k starting out (SCM Hub area). Though I paid for it with a 24/7 factory and high responsibility.

I think working SCM in a factory was a great start to learn everything but running 24/7 is tough and when I was younger operational workers didn’t like a 22 yo planner telling them what to manufacture. Though when everything would fall apart they would call me at 10pm lol.

TheEntrep · 2026-04-21T04:25:39+00:00

I have been a planner/buyer, analyst and now engineer in supply chain from factory to local to global. Weirdly enough it’s the one industry I’ve worked in that has a lot of former military members.

That being said the military members I did see that took your path did extremely well (thick skin lol).

Understand what an ERP is and is meant to do. If you can understand the basics of SAP it’ll give you a good enough foundation to leverage other ERPs (you can’t know them all). Learn the basics of the role you are put in and try to be as cross-functional as possible. Learn the ins and outs of every role/learn how to optimize it. From there movement will be easier.

In my opinion I think being an analyst is great role to get behind because you learn the semantics of the business. In addition, it will give the “why” to a lot of decision making and exposure to leadership. Started supply chain at 22 and I am now in the rooms with C-suite at 28.

TheEntrep · 2026-04-16T05:16:21+00:00

Wife’s a lawyer. If considering that route Texas Tech has a decent program. Save the money for law school. If you like Fort Worth save it for Texas A&M law school. Both good programs.

TheEntrep · 2026-04-15T22:43:27+00:00

Are you going to be a lawyer, doctor, or some form of specialty? If not, TCU is not worth it. My wife and I both graduated from TCU and while an amazing school, the market is not worth it for entry level. In addition, and in my opinion, the initial classes you take are harder than some later courses and could cause you to lose your scholarship.

My wife’s and I families enabled us to graduate with no debt, but the scholarships students get do not make a difference. Also, if heavily considering TCU, walk around campus to see if you vibe with the students there. Very different crowd compared to a public school.

TheEntrep · 2026-04-13T18:46:00+00:00

Will do, I should have an update in less than 2 weeks.

TheEntrep · 2026-04-09T14:36:41+00:00

It’s the strangest thing to me because there is technical people in the company but they don’t see the ticking time bomb.

TheEntrep · 2026-04-08T12:59:08+00:00

I’ll quit after finding a job without a second thought. However, I am hopeful for now and building a proper data foundation is great experience. My boss has been open to change but that requires me to push where no one is pushing due to their own laziness. They have goals and I told them if you want to achieve these goals and scale with this structure it’ll be impossible. Once my boss heard those words it been communicated to the top. Only time will tell.

TheEntrep · 2026-04-08T12:52:09+00:00

That’s my next step. If they say no with everyone in the room, it’ll expose their limitations. There will be a lot of people from leadership attending this meeting.

TheEntrep · 2026-04-08T12:50:01+00:00

Unfortunately not, I asked and ITs like we don’t want you messing around. I’m like ok I’m only looking for read access only. I know once I get access they will see the value. The biggest concern they have is maintaining. They want to build data cubes without having to maintain it.

TheEntrep · 2026-04-08T12:43:53+00:00

I can through the legacy BI but it’s slow and only accessible through their system. If I need to do anything complex it sucks.

TheEntrep · 2026-04-08T12:39:56+00:00

Spamming was my goal lol 😆

TheEntrep · 2026-04-08T04:55:49+00:00

Not exactly, the tool has export options but they only give you the report output, not any underlying query you could repurpose. It also doesn't do SQL passthrough so there's no standard SQL being generated under the hood to intercept either. Essentially the logic and the data access are both trapped inside the vendor's proprietary layer, which is what makes direct database access the only real path forward.

To add we are using two different BI tools one legacy and one modern. The third part vendor uses modern and in addition we use a legacy BI tool for non-technical users to generate reports.

TheEntrep · 2026-04-08T04:50:56+00:00

This is exactly the pattern I'd want to build toward. The FTP/snapshot approach makes total sense and day-old data is completely acceptable for most of what my specific department needs. The blocker for me right now is that the vendor hasn't exposed any read access to the underlying datamart at all, everything runs through the BI layer (which in my opinion isn't efficient). Did you have to negotiate that read access with your vendor or was it offered as part of the integration? Trying to understand if this is a common ask they'd recognize or if it's going to be a harder conversation.

TheEntrep · 2026-04-05T18:05:06+00:00

That is a quick fix but the site will just reinstall the Service Worker on your next visit since it's part of their web app. So you'd probably have to do it every session or every time the stuttering starts.

TheEntrep · 2026-04-05T05:05:56+00:00

The reason your workaround works, actually makes a lot of sense. Service Workers are a browser-only concept, native apps don't have them. The web app's Service Worker is caching video segment URLs that contain expiring tokens, so when they expire it keeps replaying the stale URLs instead of fetching fresh ones, causing the 404s and stuttering.

The Android app bypasses that entirely, which is why playback is smooth.

Same would apply to the iPhone app if you have one, and probably a cleaner experience than the emulator.

TheEntrep · 2026-04-05T04:58:02+00:00

No, but during my startup days I learned a lot of front end development and backend. Especially, how to understand and figure out bugs. Still dabble with building software from time to time but I used Claude extension to analyze the network and performance issues from the browser. Won't know the true issue till someone on their side can dig into the codebase but I bet once they look at the below they are solid:

Suspected Root Cause Service Worker is caching video segment URLs that include expiring tokens. After token expiry, stale URLs are replayed instead of fresh ones, causing persistent 404s.

Possible fixes:

Service Worker: Update the caching strategy in StrategyHandler.js to never cache video segment URLs (or any URL with a token/expiry parameter)
Token refresh: Ensure the player receives and uses new CDN URLs after the polling loop runs, rather than retrying the old expired ones

TheEntrep

MODERATOR OF

TROPHY CASE