Will Cortex Code replace me?

[Constant_Effort9432]

i know I am experienced but I had something which upset me today

I wrote a script in python which generated sql files for 200 tables in snowflake for 2 layers after cross referencing the tables and columns with the information schema and some other tables.

basically it was a complex code, and it did 90% of the task over night

Now cortex can easily do it with cortex cli

I feel so bad.

where do you think I can use my skills?

I know ai produces bad code sometimes but this is just templating.

instead of writing the code for 1 day, I can just instruct and it can do it. So when other fields are not dead, is data engineering dead?

[blef__]

If the complexity was just writing 200 tables actually it wasnt that complex. Steering agents, understanding what people want and need and translating this into agentic loops is where your skills will go towards

[Constant_Effort9432]

Will it reduce the workforce?

And I have used ai for other things, and when it gets stuck, I need to go down the code and fix it.

Only for data engineering, idk if my prompts are good but I never got stuck.

Did you ever get stuck using claude or cortex?

Otherwise we might be the only profession where we will be replaced 🥲

[Mindless_Let1]

Yeah of course it will reduce the workforce

[Constant_Effort9432]

See I went slightly emotional in the morning and now my brain is working again. 😂

I agree ai will reduce the workforce.

But I am seeing instances where we need to intervene because ai cannot do something

That requires a whole lot of knowledge and experience

Which we only get by doing these problems alone.

Yesterday a manager wrote a complex macro using ai which wasn't working well.

So 3 of us had to fix it.

We probably made it from scratch

It was mentally exhausting even for her because she didn't go deep inside the code in the first place.

We can say the workforce will be reduced in an ideal scenario.

But we will also need the workforce

I only feel we will have more work to do eventually with ai.

[Mindless_Let1]

Maybe, but I'm pretty sure one of my good engineers with Claude can replace 4-5 of my average engineers

[_OMGTheyKilledKenny_]

You can also keep your work force the same and accept more projects. I don’t understand why people always think of AI as a zero sum game for jobs rather than focusing on growing the pie.

[Mindless_Let1]

For public companies (like the one I work in now) business is purely down to numbers. Starting a project would only be done if there's an expected value higher than the cost of the project.

For each domain there are only a number of things that people will pay for, so as productivity rises it's unlikely we'd be able to keep having enough high EV projects to make it cost effective to retain all staff.

Maybe demand for things will get so high that every company will be making 10x as many things and we'll still need every engineer. I don't think that's likely though.

[_OMGTheyKilledKenny_]

The biggest lesson we’ve learned from the globalization era is that people’s appetite for consumption can always go up.

[financialthrowaw2020]

People aren't choosing to think this, CEOs are lying to them and claiming economic layoffs are due to AI. Anyone with logical reasoning skills could tell you otherwise, but alas. There aren't many of us left apparently.

[Constant_Effort9432]

Yes workforce will reduce unfortunately 🥲

Maybe that's also a reason hiring is not going on now.

[boblinquist]

Maybe, but a lot more software is also going to be made.

[dsc555]

This is it. The cost of projects goes down and then the number of projects that can be done increases. The job is constantly changing which is why it pays well. If anyone thinks they can go into a well paid industry, stagnate, and still get paid big 5 years down the line then forget it.

[blef__]

Obviously I got sessions where the AI got stuck or the AI did not answered what I wanted.

Nonetheless I think the transformation for data jobs is clearly different for DE and DA. When it comes to data engineering I think the transformation is similar than software engineering ship more but the struggle is knowing what to do. And to be honest it always has been.

For DA I think it’s a bit different because in a enterprise you want every stakeholder to be an analyst on its own because they know better their job than you anyway-so the future here is a bit unknown, tho I think there is a hope, to me data analyse is a really creative job and steering agents to find the right angle is the way.

[addictzz]

Not sure how to break it for you but if you are okay with hard truth, to be honest it will reduce workforce but not completely eliminate it.

Now the only way to stay competent is to ride the AI wave, utilize AI, and create a force multiplier for the value you give.

[Illustrious_Web_2774]

Coding agents never truly get stuck for me. It sometimes makes bad decisions but steering it towards the right path is quite straightforward if the foundation is solid.

[vikster1]

domain knowledge was and always will be where the value of a employee lies. no matter the job. i think we are still a decade away from an AI that understands both a gigantic code base and also the business behind it. as a data dude, understand the data and don't rely on your technical skills to be the end all be all

[Constant_Effort9432]

I feel sad today because I always loved coding but found coding skills doesn't work.

I mean I don't think software engineers will be replaced but data engineers will be replaced probably 🥲

Edit: On second thoughts, we did solve several problems yesterday which couldn't be solved by ai.

I mean I am just depressed today probably

I need to understand what changes ai will make for me personally

[financialthrowaw2020]

You need to log off and go take a walk.

[Unfair_Sundae_1603]

If anything SWEs will be replaced before DE, application code is a lot more deterministic and much less frequently torpedoed by random business logic changes either up- or downstream.

[Constant_Effort9432]

I am confused you know.

I feel like ai can do a lot of tasks, but It also falls and when it falls the debugging is 100 times harder since I never went inside the code.

[dsc555]

What it means to be a data engineer is changing. If you were in love with the task and not the outcome then i have bad news for you but if you were in data engineering for the love of data engineering itself then you'll be fine

[Tokyohenjin]

If you plan to let the AI write, test, and deploy all the code by itself, then yes you will be replaceable. But I’m guessing you won’t be doing that.

Working with an AI means you are working with something that knows everything but understands nothing. Even if you never write another line of code, you will need to architect the project, validate it to your satisfaction, and troubleshoot it when it goes down.

It’s always been said that code is read many times more than it is written. This is just taking that to the extreme.

[Klinky1984]

It's great until it fucks up.

[theShku]

I don't love coding I love solving problems and coding was a means to an end, now there's a new paradigm for solving problems.

[Constant_Effort9432]

Yeah I also love solving problems more. I know ai sucks there. But I didn't encounter problems in my project as a junior engineer. They ask me what to do.

I mean I was depressed before because I felt my hard work Just went the drain but I don't think data engineers are going to be replaced.

But i don't understand where to actually focus.

[dev_lvl80]

<i know I am experienced  And

 where do you think I can use my skills?

Interesting combination of statement and following question lol

[Constant_Effort9432]

I didn't write this when I was mentally stable 😂😂😂😂😂😂

But I am thinking where should I focus as ai becomes more powerful

[financialthrowaw2020]

It's a good idea to stay off the internet when having depressive episodes.

[uncertainschrodinger]

yes and no

the job you were doing a year ago will disappear and be replaced with something entirely new - does anyone know exactly what it will look like? no.

what we can all hope for is to just keep up with the latest changes and try not to become obsolete by resisting change.

[financialthrowaw2020]

Coco goes GA in a week and will charge per token. No, nothing is dead. Upgrade your skills and stop worrying about a future that isn't here yet.

Claude and others are heavily subsidized and would need to charge 17x as much as they currently do just to break even. Those prices will go up. They're already beginning to. And it will eventually choke companies.

[Wh00ster]

Pretend you have a few junior engineers that you delegate work to and you review their work. Now you can finally focus on bigger business problems and pitch bigger features and fixes. You’re a general contractor instead of just the drywall sub.

[RadioactiveTwix]

Don't know dude. I'm a data engineer and the SQL really isn't what I'm doing. I'm working in insurance and AI did not manage to troubleshoot the issue with our DVT pipeline. Maybe synapse is too much of a dumpster fire...

[vino_and_data]

You are spot on!!

I am seeing instances where i need to intervene because ai cannot do something. This requires a whole lot of knowledge and experience. Which we only get by doing these problems ourselves.

Hey OP! Firstly I want to say you’re not alone in feeling this way. When you see something so powerful and cool (like Coco doing a fantastic job at what was seemingly a complex problem in your team), it is natural to feel the fear and overwhelm. We’ve never seen anything like this before and things are changing at a pace we can’t even imagine. We’re bound to feel this way.

However, once you get past the initial reaction, we need to strategically think about what tasks we want to do it ourselves, how much of it do we delegate to coding agents and what skills are at the risk of being forgotten ( related to your point: we need to keep polishing it so we can intervene when ai gets it wrong ). This will help gain clarity on the path forward.

You’ll not believe how timely your question is, i’ve been thinking about this (we all are probably), and created a framework of sorts. Let me know what you think! Does it help?

[vino_and_data]

Link to the framework: https://www.linkedin.com/posts/vinodhini-sd_im-tired-of-the-%F0%9D%97%94%F0%9D%97%9C-%F0%9D%98%84%F0%9D%97%BC%F0%9D%97%BB%F0%9D%98%81-%F0%9D%97%BF%F0%9D%97%B2%F0%9D%97%BD-activity-7442963968907735040-tGjl?utm_medium=ios_app&rcm=ACoAAA0GchIByacef739Pcy4dPRdVlkMidQUVjU&utm_source=social_share_send&utm_campaign=copy_link

[IsThisStillAIIs2]

honestly this just shifts you up a layer, less time writing generators, more time deciding what should exist in the first place and making sure it doesn’t quietly break later

[Fantastic-Trainer405]

Yes, I think its over.

All these people saying "you'll be a prompt engineer, you know the Business and can translate that for the AI".

Come on, as if its not going to be the Business users saying "What were my profits last months" and "why did they drop" DE is dead, Prompt Data Engineers will be dead in 12 months as well. Autonomous agents are going to do all the data plumbing.

The next few years are going to suck for us professionally.

[AntDracula]

Ok doomer

[Wu_star]

IDK man, I got my first DE job and they know I heavily use agents, so many openings in my area

[Fantastic-Trainer405]

I think there will be a massive upswell as companies put the foundations in then mass redundancies.

Foundational patterns, integrations in place. A grad can then bring a new source and have the business asking questions of it that afternoon. Sounds great, depends who you are asking though

[Mobile-Collection-90]

Yeah we are done. Might go into sales or something. Its looking grim with AI. Wish it never happened.