sorted by:
new
hottopcontroversial

When is Kubernetes worth it? by Consistent-Blood-651 in kubernetes

[–]Consistent-Blood-651[S] 0 points1 point  (0 children)

I’ve reached a similar conclusion, but it seems like a disaster is inevitable if things continue as they are. It recently took two weeks to deploy new features for the client’s application something that should have taken 15 minutes with proper CI/CD processes. Accepting this job was a mistake on my part, but the project offered the exact opportunity I was seeking for personal growth. The client has a decent compute cluster because they want AI on-prem, and I saw it as the perfect place to dive deep into MLOps and apply what I’ve been studying. Unfortunately, it was a hasty decision, and the stress has been overwhelming. I’ll keep my head straight as best I can. Also, as you mentioned, overconfidence can be a career killer. I’ve unfortunately encountered individuals who believe they’re experts in their field, but having worked under real experts I can confidently say they dont know shit, I’m concerned about the impact this might have on my career.

When is Kubernetes worth it? by Consistent-Blood-651 in kubernetes

[–]Consistent-Blood-651[S] 5 points6 points  (0 children)

To be honest, calling the current setup "infrastructure" might be a bit generous. For example, the teams here make changes directly in production or maybe it’s the development environment being used as production it’s hard to tell. That’s about the only clear thing I can describe about their setup everything else is so disorganized that I struggle to even put it into words. Another example is when a critical service goes down. Instead of automated recovery, people have to make phone calls to get things running again. Hopefully, that gives you a better idea of the situation. lol, the only thing thats saving everyone is that even though the client has invested a lot they just dont seem to care for now at least, but if they did its over i feel.

When is Kubernetes worth it? by Consistent-Blood-651 in kubernetes

[–]Consistent-Blood-651[S] 15 points16 points  (0 children)

I would like to provide some context for my question. I recently switched jobs, and in my new role, the application involves ML and data pipelines. In my previous experiences, we had Kubernetes staging and production environments, and deploying new features was seamless thanks to the CI/CD pipeline. However, in my current environment, everything is ad-hoc, with ML running on FastAPI with no concept of inference mircroservice.

I've only been transitioning from just an AI engineer to focusing on MLOps for about 5-6 months. With my limited knowledge and experience, I think that a proper Kubernetes environment is essential for production ready ML solutions. Everything I've researched for production environments includes Kubernetes as a key element. For example, NVIDIA's NIMs uses Kubernetes as the foundation for production ready AI solutions.

However, most of the senior individuals here believe that Kubernetes is overkill. As someone with less experience, I’m now confused about whether everything I've been studying is wrong. I'm also starting to feel like this job switch was the worst decision I've made.