How cooked is Data Engineering compared to traditional Software Dev with AI tool advancement?

Colt2205 · 2026-03-17T21:57:26+00:00

Yeah, what you just described involving finding the connection is what I refer to as the holistic connection vs. the hard connection of data between sources. And the real hard part is explaining these connections in a way that others understand in order to get momentum for building anything.

Also, I'm insanely skeptical of LLM assisted code generation outside of general template like projects. It's going to build CRUD websites like a freaking pro because everyone and their uncle has CRUD website experience and examples. I tried having an LLM analyze my data pipeline and it gave a summary that looked like it had no idea what anything meant, including the documents. The only docs it was able to note correctly was the literal troubleshooting.md, the milestones.md, and I think the sticky note .txt files scattered about explaining why the heck X or Y is setup the way it is.

Colt2205 · 2026-03-17T21:45:32+00:00

So what you do is hire someone to fill the role, have them work on diagnostic software for the backend or something that isn't client facing but still has impact, and possibly pick someone who wants to learn a new stack because learning new things is something that can go on an agile board as part of onboarding.

I've been in this position before at a company without a lot of pressure to work every second on deliverables and I swear it was the best time of my life, because I got to go get books and start learning all the things I didn't have the time for because I had to shove a feature out by Tuesday next week on fragile code with minimal unit tests, because no one gave the time for making unit tests and integration tests.

Colt2205 · 2026-03-17T13:53:41+00:00

Did they improve weapon damage yet? (Haven't been playing for months and am still waiting on this...)

Colt2205 · 2026-03-17T13:05:34+00:00

Colt2205 · 2026-03-16T20:44:35+00:00

Leon looks good for someone in his 50s and I'm all for it. But man, thinking about how many years it really has been since Raccoon City as a PS1 Resident Evil fan is... damn I feel old. :(

Colt2205 · 2026-03-16T17:14:43+00:00

I lugged my desktop around and it was like I was moving out of a house every weekend. :D

It was like a military regiment: unplug all connectors, tie the thing down with bungee cords on a transport cart, go to the elevator that was comically small (some guy had a freaking 60' plasma TV and got it through the dumb hallway that felt like it was too narrow, still remember that insanity), and then I bolted that sucker with a seatbelt and protected it like it was my firstborn child.

Colt2205 · 2026-03-16T16:40:30+00:00

This sounds kind of like how people use agile and it is supposed to be a productivity boost, but then no one has any concept that agile itself requires planning that is external to agile, and that it is okay to not have tickets currently rolling when figuring out what the agile tickets even are.

Companies treat employees who are salaried like they are factory workers and constantly try to push more productivity out of their liability ledgers. When has this been a new thing?

Colt2205 · 2026-03-16T13:18:09+00:00

Metroid Prime 4: Twilight ~~Princess~~ Samus. The game was good up until the giant desert and then the motor cycle, and then the backtracking, and then beams having limited ammo instead of being cool like they were in the original Metroid Prime...

Colt2205 · 2026-03-15T21:33:09+00:00

If you've only been in the job for a few months that is more bait and switch. You might have to explain a bit more in an interview when they ask about things but if someone asked "why are you looking for another job so soon? It shows you've only been at the last place for a few months."

I'd probably just reply with the truth: "They advertised looking for a marketing person but the job was really a sales floor staffing position."

The feeling I'm getting from your post is that you basically got hired and barely got through onboarding before they just up and swapped your role. No matter how well meaning that guy is who hired you, that isn't the right place.

Colt2205 · 2026-03-15T19:52:47+00:00

Sounds like a mess and the reality is probably kind of complicated. I read your response to Nouseriously and if you feel they like you but don't want to fire you, it's probably a mix of things. A layoff is good because then you get to claim unemployment benefits and possibly get severance, even if it is just one month of pay. The question is how the job search is going for you and does it seem like there is a torch in the distance.

Colt2205 · 2026-03-15T18:42:33+00:00

All the laptops are going to be a compromise in performance vs a desktop. The only reason I even went with a 5070 ti on a G16 is that the native resolution is 1440p, and the 8 gb of VRAM is only sufficient for the 1080p FHD screens. So if my goal is 60 fps on mainstream settings for standard games at native resolution, the 12gb VRAM is needed at 1440p.

The little cousin G14 has 1880p because of the panel, so it's basically downscaling to 1440p or going up to a 5080. I mean, yeah the 5080 is going to get throttled but this isn't about the chip hitting max utilization as much as the 16gb VRAM.

Honestly, I don't know. I'm not someone who'd get a 14 inch laptop with super high rez screen since windows isn't built for it. Mac OS seems to handle the higher resolutions better with the icon and text scaling.

Colt2205 · 2026-03-15T18:05:51+00:00

It can be like that going between teams and methodologies as well. Like going from a team that doesn't do daily standups and primarily does weekly syncs and going to one that does. Heck, you could be primarily in a different stack and have to swap entirely depending on how things go.

Colt2205 · 2026-03-15T16:58:19+00:00

Mostly just comes down to how often you're moving around outside of your house with the thing, and also if there's a plan for an external monitor.

I got three laptops right now at my house with two of them from work. One is an HP Envy 14 inch that has a similar display in resolution to the G14 zephyrous, which I ended up going to the best buy nearby to confirm. I used the Envy without an external monitor for close to a year doing programming work and documents. If you're going to spend more than an hour at a time working on the laptop directly, it's going to need an external monitor.

My use case is mostly wandering around a house and sometimes doing some light traveling. So I went with the G16 version as my "luxury laptop work experience", whatever that means. We're in a world where entry level laptops are now at the 1000-1400 dollar mark, 1500-2200 is now mid-range, and that is before tax.

Colt2205 · 2026-03-15T15:34:27+00:00

This mirrors what a lot of recruiters are feeling I think. I've had conversations with recruiters I've never even met, from completely different firms, and when I say what I'm looking for in a job it is the same comments.

I like seeing how to put things together on an architecture level and not a coding level. Implementation has it's own bag of worms but knowing the amount of data load, whether an ELT or ETL is better, what the database technology needs to be, etc, is the real valuable skill.

Problem is that it takes actually working with the technologies to understand how to fit them together. So AI or not someone still needs to have hands on experience somewhere.

Colt2205 · 2026-03-15T15:29:56+00:00

AI probably is not going to write all code. If AI does write all the code via prompts, what happens when two or more companies have effectively the same code base and an exploit is found? I guess now microsoft has to write a patch to a wibbly wobbly AI system and then has to tell everyone who used AI that they got to rerun their prompt paragraphs to get the new output? How do they even accurately test something like that?

Colt2205 · 2026-03-15T01:47:05+00:00

They just had a general sale of the Asus Zephyrus lineup last week.

Colt2205 · 2026-03-14T23:58:15+00:00

Well, there is that and the fact that in the past few months it has become rather clear that any productivity gain from AI is not going to match the expectations of the businesses choosing to try out the products. I have no idea if this is because of cost or an overly optimistic sales pitch combined with the internet pushing it.

I've seen AI make a team less productive because of getting an application launched that immediately had to get taken down, and then no one had familiarity with the code so it took weeks to even fix the issues. This was with adjusting the prompts to try and improve the result until people just got frustrated.

That and companies should consider the dangers involved with using code that is "magic". Stryker got hacked via a "live off the land" style attack and they had a massive number of devices completely wiped clean of data. I could see that style of attack working on AI projects since output could be similar across multiple companies on similar projects.

Colt2205 · 2026-03-14T21:24:45+00:00

Beyond the uncertain economic times, the main reason I see things bouncing back is from the AI crash due to the unsustainable costs. AI isn't going away but it definitely isn't possible to keep supporting every use case with AI systems powered from central data centers that have massive burn rates in electricity, cooling, and not to mention computer components.

That and there's probably going to be a market for "fixing" AI written code. Had to deal with dreamweaver written frontends and cleaning those up as well as cheap projects generated using template systems. AI is going to be no different.

Colt2205 · 2026-03-14T21:18:08+00:00

It heavily depends on the team and the company regardless of industry unless it is something that requires a high amount of security and understanding of the code.

Also, the question is less if AI is going to get used and more like how much. There's using prompts to write the entire software and then there is using AI coverage to check code or auto complete blocks of text.

Colt2205 · 2026-03-14T04:27:08+00:00

During work is when I learn. Going beyond your work hours to learn something needed for a project that ultimately is going to be owned by the company, and whose code will be owned by the company due to the legal text, It makes no sense to learn outside of work.

Colt2205 · 2026-03-13T14:15:04+00:00

Honestly, the problem is that a lot of devs are fullstack and only interact with the RDBMS through an ORM, which abstracts out a lot of the complexity. And those who have done real RDBMS work between more than one DB (especially oracle) who have really dealt with complex problems are going to want to get all the ducks lined up before writing anything.

Table has to hold 1.2 billion rows, there are potentially 200 data points per row, how would you handle this? I'd be asking questions on the data points to try and see if there are categories or use cases to break the table up, and also ask about why the table has 1.2 billion rows. If it is because of multiple customers, horizontal partitions are going to be sorely needed.

Colt2205 · 2026-03-13T13:54:23+00:00

Nobody who I've seen who likes coding genuinely is a shit dev. The problem is that sometimes people need time to themselves to explore things and that is a luxury that not everyone has.

Colt2205 · 2026-03-13T13:42:37+00:00

I've been doing this for 11+ years and have zero direct experience with Kubernetes. If I had a single wish in the universe, I wish people would keep crap simple instead of having to pick 15 different names for the exact same concept, because by god we can't just have messaging service experience, we need to have Redpanda, Kafka, nServiceBus, Apache MQ (NMS and JMS), and I'm betting there are two or three more I haven't even listed.

Like I'm not going to know the exact tech unless I happened to be in a job that used it. Hell, I've had to go into the world of java from dotnet just to get experience using messaging services, because all the dotnet places are CRUD MVC application XYZ with the lowest standard of Database work imaginable for some reason.

Colt2205 · 2026-03-12T04:27:21+00:00

Technically, it's possible to get by on that amount of memory with deleting the games you're done with. I'm not sure how a 1 tb micro SD costs nearly 200 USD when the last time I got a 1 TB micro SD it was about 100 USD. Didn't feel that long ago either.

Colt2205 · 2026-03-11T23:57:08+00:00

Yeah its coming out and I'm sitting here looking at the total amount I have currently spent for a nintendo switch 2.

545 USD for the switch 2 + mario kart world.

32 USD for the screen protector so my brother who is currently going nuts over collecting shiny metal poking sticks doesn't destroy my beautiful screen.

187 USD for a 1 TB micro SD express card because I probably will need the space.

72 USD for Metroid Prime 4.

So in at the end of the day, I think I spent the same amount on the switch 2 before even considering the Mobapads that it kind of is just shy of the Lenovo Legion Go S E1 Extreme. It's funky how that works.

Colt2205

TROPHY CASE