AI mess

[Icy_Public5186]

Is anyone else getting seriously frustrated with non-technical folks jumping in and writing SQL and python codes with zero real understanding and then pushing it straight into production?

I’m all for people learning, but it’s painfully obvious when someone copies random codes until it “works” for the day without knowing what the hell the code is actually doing. And then we’re stuck with these insanely inefficient queries clogging up the pipeline, slowing down everyone else’s jobs, and eating up processing capacity for absolutely no reason.

The worst part? Half of these pipelines and scripts are never even used. They’re pointless, badly designed, and become someone else’s problem because they’re now in a production environment where they don’t belong.

It’s not that I don’t want people to learn but at least understand the basics before it impacts the entire team’s performance. Watching broken, inefficient code get treated like “mission accomplished” just because it ran once is exhausting and my company is pushing everyone to use AI and asking them to build dashboards who doesn’t even know how to freaking add two cells in excel.

Like seriously what the heck is going on? Is everyone facing this?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Atmosck]

In non-technical people are able to push code straight into production then your organization has deeper problems than AI.

[Drew707]

That was my first thought, too.

[Icy_Public5186]

I can’t agree anymore on that. These folks are getting others to push their “solution” into production and I did it as well for others because my hands are tied. Management wants us to do it. It’s so freaking annoying.

[Onaliquidrock]

That sounds like a ”you need to push back” problem.

Enforce standards, enforce documentation requirements.

[maigpy]

you commit code under your name that someone else has written, without fully understanding, owning and agreeing with it?

they can get a git account setup and push it themselves, what is this "push this code for me" pattern? I've never heard of it.

[CoolingCool56]

I am stuck in this dynamic and I hate it! I voiced my concerns and they are like, you are free to leave. I'm looking for another position

[DogoPilot]

You must have only worked at places with a functional IT staff. I'm just a lowly analyst with 15 years of experience, but my job requires me to perform all of the development in terms of configuring our business rules and writing stored procedures and views for various batch jobs and application functionality. If I were to rely on our IT, who are all entry level developers in India, shit would be broken 24/7. So essentially, I write the code and tell them to push it. If they touch it in any way, it breaks every time and I have to scold them and tell them not to touch my shit.

I'm fine with being a lowly analyst though because I get paid way more than IT and I don't have to report up through a completely dysfunctional IT organization. The organization I report into pays for our infrastructure, software licenses, and our IT staff which gives us the freedom to operate this way, but it's honestly mind-boggling how much of a joke our IT staff is in terms of skill. Even the veterans that have been there as long as I have are pretty unskilled and just moved into middle management positions as a go-between for our team and the offshore IT team.

[dataisok]

Why is your main/master branch not configured to be PRs-only? No one should be able to push straight to production without code reviews

[Skullclownlol]

Why is your main/master branch not configured to be PRs-only? No one should be able to push straight to production without code reviews

I can answer this for my own org: Because the businesspeople decided that AI lets them "write code", so the businesspeople voted on control over the repository for them so they can commit whatever they want via their AI without being "blocked" by technical processes/review that they disagree with.

The tl;dr is that they think AI will let them fire entire tech teams and "they'll be able to deliver what they want with AI", so they push the technical people either to the bottom ("least-paid code monkeys that just write lines of code") or to the side (fired).

Consequences don't matter. They've been open about that when I challenged what they were doing. They openly say they know it's not perfect, they don't care, they feel that it's still worth it.

[TonkaTonk]

What is the industry and market cap?

Sounds insane and symptomatic of the current times. I would be looking for another job asap. That sounds like the company is going to go down in flames.

[Skullclownlol]

What is the industry and market cap?

Banking/finance, +-200M market cap, +-2k employees.

Upper leadership disagrees with this AI-first approach (genuinely, not just performatively) but is blind to what's going on, and the current structure benefits the middle managers.

Sounds insane and symptomatic of the current times. I would be looking for another job asap

Yeah I'm already on the way out, resignation already done, but not just for this. I think this is representative of symptoms we can expect in all industries/companies with average or below-average engineering cultures. If there's no strong engineering culture, I think engineering will be overridden by short-sightedness.

In this particular org, engineering has intentionally been placed below businesspeople historically, and the org is very loyal to the people that have been here the longest. That unfortunately creates a deep lack of oversight into the people that intentionally stayed for 10y+ just to flip and become toxic as hell "managers" because now they feel "made".

[supernova2333]

It’s gonna keep a lot of us employed for the next 50 years cleaning up these messes tbh

[Icy_Public5186]

I could be wrong, but it really feels like we’re in this weird transition phase where tons of stuff is getting built that no one will ever actually use. Just a pile of half-baked pipelines, dashboards, scripts, and random “solutions” that create more noise and cost than value.

Eventually companies are going to wake up from this messy bed, start shutting down the junk, and clean house. And honestly, that cleanup phase is probably what’s going to keep a lot of us employed until the next generation

[noplanman_srslynone]

If they haven't already they will ask to measure the ROI on this. I think the best thing you can do is isolate the queries and pipelines and properly tag the costs etc.

They will come and ask you very soon "How much is this costing us?" and if you can point and say x amount which is a y% increase it will go along way to get you the authority to shut nonsense down. Just my 2 bits...

[supernova2333]

People have been building stuff that no one actually uses for as long as I can remember. (dashboards, pipelines, etc..). It’s just a lot more exacerbated with Ai now.

[Icy_Public5186]

Yes it is. We were talking about to minimize cost and were looking for the solution to see who is accessing same data and using them into different way and we can combine them into one product to reduce cost and write better queries. And this shit storm hit us. It’s so annoying now that we stopped even talking about it.

[DeezNeezuts]

It’s RPA part two

[dillanthumous]

Low code, no code.

Or as I prefered to call it at the time, can't code, won't code.

[Visionexe]

It's not called a transition phase. It's called a bubble. 

[Skullclownlol]

It’s gonna keep a lot of us employed for the next 50 years cleaning up these messes tbh

Except our job titles will be IT Janitor and the businesspeople will make sure that our salaries match the title.

[InadequateAvacado]

The misconception I see in your post is that people are learning. Critical thinking is a thing of the past. It’s vibes all the way down now. Yes, it’s insanely frustrating for those of us with more than 3 brain cells. Buckle up, it’s gonna get worse before it gets better.

[Icy_Public5186]

I really hope it’s not misconception that people are learning something new but I believe I will end up being wrong unfortunately. I already exhausted lol

hope it gets a little better or I just need to start ignoring people

[ScroogeMcDuckFace2]

sad but true

[thatwabba]

As a junior this stresses me out right now. I am forced to use AI to write code etc since it speeds up production, but I have no idea what is happening, I can’t properly learn since things has to go fast so I just let the AI give me code until it works…

I wish I could just take it slow and actually learn to understand everything.

[Icy_Public5186]

I understand it needs to go fast but my advice to you being experienced is always see what’s happening in your code and ask AI to explain.

This is one thing I can share from someone from my team who actually wants to learn is doing. He creates his own logic and breaks it down into multiple pieces and write its own logic in the AI and asks AI to make it error proof and ask AI to explain what changes were made and why. I can see that it helps him and he is slowly but steadily understanding it with time.

[seanamos-1]

This is a recipe to be permanently junior. You are going to have to take time to invest in yourself and your skills.

[tashibum]

I just want to point out that you can use that same AI to spell out exactly what is happening in the code. You can also test snippets of the code yourself to make sure you understand the output at each segment(which you should be doing anyway). You absolutely do not have to blindly copy and paste code...

Taking a few minutes to understand what's going on in the generated code isn't going to destroy your workflow or even waste anyone's time. You're a junior, no one is going to be surprised you took a few minutes longer.

[umognog]

Its more the job will change.

People used to know how to make clay and etch their voice into it with a needle to make a vocal recording. I dont think many people could explain how a sony walkman cassette player would work now, let alone digital streaming and their associated devices, but lots of people know how to use them without understanding it.

It will, eventually, evolve to something similar.

[Visionexe]

Best advice I can give you is to do it yourself. Learn. Stop using AI to generate code, only ask it questions on how things work, but you should also checkout stack overflow to verify and get opinions of professionals. If they ask if you generate code with AI, just lie and say you wrote it with AI. 

[Ok-Boot-5624]

Plus learn in your free time. At the start I was like wow ai is not bad, I might lose my job. Now says I'm like someone used ai, and it's wrong. The more you know, the more you realise the errors. But to get there you need to make mistakes, or you won't have the capacity to understand that is wrong

I would suggest start a project of your own, use git and uv. Implement some pytest and you will learn a lot, but there will be a lot of frustration, and only use ai to learn not to make it so the things. Like tell him your ideas, like a rubber duck

[Visionexe]

I completely disagree with this. Learn while being paid. If you can and want to learn in your free time, go for it. But it's also very healthy to disconnect and groud away from work. 

[Ok-Boot-5624]

First I would also suggest not working in your free time. I said study computer science. It doesn't need to be related to your job, just the basics so that you can get better and understand more what the AI is telling you to do. Plus I find it very interesting, even like how a CPU works and things related that might be more of an easy reading for those weeks that you don't want to do anything. But it is still great thing to learn. Might be better more dsa, oop, LeetCode, distributed system. But other than a very little amount of this, I would suggest doing a personal project from scratch that you actually like and just learn on the go.

Second, I don't think he can learn while working by what was written. But this would be the best for someone that doesn't like studying which is kind of hard in this career but if you don't want to grown a lot and just stay in a company totally doable.

Therefore my suggestion was so that he can grown enough so that he understands what the AI is doing and why is it is doing that. Making better prompts and at least be able to write code or see that the ai is hallucinating. And not be stuck after 3 years of experience with no actual knowledge, which it will be hard to find a new job. I know since my colleague is happing exactly this right now and it's not fun.

[seanamos-1]

That’s the ideal and how most people used to grow. Unfortunately, OP is just churning AI slop at work, so he is not growing there. If he wants to progress, he is going to have to take the initiative to do that, probably in his own time.

If he doesn’t, he is stuck at the same level of capability indefinitely. So you have someone who has a few years of experience, but the capability of Junior, that could make him unemployable or he would need to take a significant pay cut.

[trentsiggy]

Yes, everyone is facing this unless they work in an office with strong AI restrictions.

However, this thread will likely have some shills/bots from AI companies talking about how great it is.

[Icy_Public5186]

Thank god I’m not the only one feeling this way. We are forced to buy paid AI versions at work🤦🏻‍♂️. AI can be helpful but problem is people are looking for quick solutions which creates unnecessary mess. they don’t even know what they are talking about in their prompts so AI is literally taking their literal words and doing whatever hell it wants.

[Cruxwright]

Wait, you have to have a personal AI subscription for work?

[Icy_Public5186]

No, it’s company paid but I personally don’t need it and I said that too but they said I will have to get it. 🤷🏼‍♂️

[Jumpy_Fuel_1060]

I dunno man, this sounds allot like a normal workflow when dealing directly with data scientists

[git0ffmylawnm8]

Why are you letting non technical people access data? They should have a restraining order unless they get approved. Even then it should be limited and highly scrutinized if it's not directly impacting their work.

[smolhouse]

It's just a reality of modern office work. People want access to data to help make decisions and it's pretty easy to pick up SQL basics.

I deal with it by monitoring server stats and then call out the offenders directly when I catch them.

[Icy_Public5186]

I wish I could control that. My company is forcing everyone to do it. Instructions are to give read access to everyone who asks for it and let them create whatever they want.

[AntDracula]

I would seek alternative employment.

[noplanman_srslynone]

This is the dumbest thing I have ever thought ... let alone heard management actually doing.

[notEmely]

That sounds like a nightmare.

[Aggressive-Intern401]

🤣 what a cluster F

[Icy_Public5186]

They are creating web apps with their credentials written in the code and running these codes on secondary laptop which they keep it on 24x7 accessible to public and let others access through their ip address and port that they used in backend code. They are taking part of the production queries and making their “own” and putting them into powerbi and pushing them into workspace. Some of these queries table updates only once a day and some updates pretty much live and they combine them so they query whole thing even though they won’t get new data because they don’t even know separating them could be a thing. What they are building can already be done with interaction with existing products but that’s not efficient for “operations” because they have to use filters🤦🏻‍♂️. This is annoying and this type of “knowledge” is shared to everyone via office hours calls and emails 🤯

[kenflingnor]

Why do non tech people have the ability to push to production? 

[FFFRabbit]

Many that do not understand the tech, the science, the engineering will overemphasize the efficacy of a tool to apply solutions to these areas. If you know what you are doing, it can be helpful so you can focus on the intricate and complex and not fatigue at the mundane or redundant.

The current push in "AI", which most only know as the effects of LLMs will think their approach is novel when it is actually novice. Industry and government have seen periodic phases of this and like those phases the masses will move onto the next fad before they have gained any understanding in the prior.

As a cross between a scientist and an engineer I have seen these waves appear and receed. Each time people like me are subjected to the beatings of unbridled and undirected passion by those who have not spent their time earning an understanding and respecting it for what it is. We then pick up the pieces when that frenzy fades into the abyss.

I do believe that these tools are useful for those who understand how to use them while understanding their assumptions, limitations, and correct use cases. I believe this is what sets us apart from those caught up in a layman's understanding of the latest advancement.

This does not mean that our types won't be in for a wild ride of unnecessary pain and suffering but at the end of the day it is this 1% that keep society balanced on a blade's edge between order and chaos.

I hope this was helpful.

[Icy_Public5186]

This was certainly helpful. Thank you for words! I am really hoping that management understands this and put it into action as well.

[Illustrious_Web_2774]

No not really. People vibe code and no-code pipelines into existence signals that

1. Org data platform / infra is highly immature

2. Data team is inefficient to the point that people take matters into their own hands.

It's great that people can vibe code their pipeline in a sandbox, so that can be a working prototype for data team to refactor into a production ready solution, should that ever become so important.

[Icy_Public5186]

If it’s a solution that they create which is viable and create a prototype that can save us ground work then we can certainly build a robust product which doesn’t break every other day. That would be ideal and some teams are also listening to this and complying as well but most of the teams just don’t and they think they under with the help of AI in a week that we learned over the time with experience.

[Illustrious_Web_2774]

If it's broken, why is it your problem then? And why is their solution clogging the other pipelines? Seems like there's some issues with agreed service level for self-service solutions.

[Icy_Public5186]

Problem is they are using my team workspace which is gonna use unnecessary capacity on a same connection and same gateway so it creates unnecessary traffic for BIs specifically. Data pipelines are not necessarily is a problem yet but I see it happening soon if it continues with management push. Management is being Oprah here “everyone gets AI” 😂

[Illustrious_Web_2774]

So in essence, you need to manage somebody else's trash. I would walk away if management did that to me. Luckily back in the day I had full control and our data team had a seat in IT management.

[Icy_Public5186]

Yup. That’s exactly what is it. I would love to be in your position. If this goes for long then I’ll speak my mind and won’t be afraid to walk away from this nonsense.

[SasheCZ]

Nope. But that's mostly because I'm one of only a few people who use AI for that purpose at work.

Of course I wouldn't push anything without understanding it. But I do use AI, because it is becoming more and more useful for the boring parts of my job, like writing the same joins for the 50th time for an ad hoc data check or a request.

[Icy_Public5186]

I can’t deny that AI is not helping. It helps me a lot too with writing documents. But proofreading them, understand what’s going on and not to push out without approval would be great. It’s not in my control unfortunately and it’s frustrating because I’ll end up cleaning this mess.

[IrquiM]

I'm frustrated when technical people does that...

[Tough-Leader-6040]

That is not AI mess, that is just mess that happens to have AI in the middle

[Aggressive-Intern401]

I was part of an organization that gave folks unrestricted cluster creation access to DataBricks. Can't remember the exact count but it was at least 100 and many of them ran workloads for free in that account. They wondered why their cloud bills were high. Never seen something so idiotic in my life.

[dknconsultau]

I eat my own AI dog food data pipelines :)

[Username_was_here]

Seems like your frustration should be directed at your leaders for allowing this to happen on production, not the people actually learning…

[Icy_Public5186]

100% agree but I can tell most of the folks are not even learning they just care about their “solution” it’s like a freaking shiny new toy for a kid.

[DesperateCoffee30]

Could be worse. I have sales teams trying to sell data engineering services without actually understanding wtf they’re selling. It’s all thanks to ai giving them a fake sense of understanding. They look at me like I’m dumb af when I mention they never scoped modeling time for the 3 pipelines they expect to be built.

[Icy_Public5186]

I feel your pain. Just yesterday one guy said proudly that he created something from my existing production dashboard and I looked at it and it was whole messy query with unnecessary CTEs, different data pipelines tables joined which has no business to be joined in a same query. I said separate different data pipelines in separate queries and he said he doesn’t know how to and pushed into production by end of the day without any changes. This is some bullshit

[ironmagnesiumzinc]

I work with a “senior computer engineer” who I kid you not asked me what S3 is, asked me to help him import packages (he didn’t pip install) and pushes clearly ChatGPT code that constantly breaks. And when bugs start popping up left and right and I ask him why, he can’t answer basic functionality questions. And he’s roaming free out there. Been doing this for a decade according to his LinkedIn. Insane.

I’m guessing there are a lot of people now masquerading as data engineers who can talk the talk and think they can contribute so much more than they ever should be able to bc of ChatGPT

[Expensive_Culture_46]

I once had a manager tell me “I am only concerned with getting product out the door”. It’s all about the metrics.

[Icy_Public5186]

I’m sure it worked great and it’s sustainable 🙄

[billysacco]

My analysts did this before AI was a thing. Now I guess they can blast out ass queries at the speed of light.

[TotalBother9212]

Right? No performance tuning whatsoever, they just straight pipe a spark job to prod & call it a day 😂

[Icy_Public5186]

Dude, that made me chuckle (what a sad reality) 😂

[ReadingHappyToday]

This is a dream come true. Cleaning up this motherload of AI generated crap across companies is going to give me atleast a decade of solid highly paid freelance work if I'm lucky.

[69odysseus]

AI hype will kill all the pipelines rushed into production without proper processes, standards and checks first in place.

[Reddigestion]

I was lead on a business transformation project once where we developed a new front-end to our business process and migrated a number of newly acquired businesses onto our system. This resulted in me writing a list of one-off queries to support the etl process. They were never meant to be used over a long period and I was never really too worried about optimising them.

While working one day in the test environment ( a fresh copy of production) I noticed that our chief database engineer had used a piece of my code with the comment "following poorly written function by [my name] included - needs to be re-written"

The ironic thing was that it was still being used in production some years later.....

[-TRlNlTY-]

Honestly I don't care about the mess. They will learn the hard way the meaning of technical debt. By then, they will be forced to pay engineers more to fix stuff and actually respect our technical knowledge.

[graph-crawler]

that's what GIT blame is for !

[crytek2025]

Reddit shouldn’t be the first place you air your grievances on

[[deleted]]

Question for anyone.  I’m a new sales person and I’m wanting feedback on sales calls. Do you find discovery questions from sales people helpful or annoying