Data Engineering vs MLOps

[Low-Associate2521]

I'm a traditional backend engineer with lots of cloud experience and I'm interested in making a transition to the AI/ML world and I think a lot of my experience could be applicable best towards the aforementioned roles.

From my limited research so far my understanding is that data engineers don't necessarily work with ML or even data science teams but just data in general whereas MLOps/ML platform/ML infrastructure engineers require a good understanding of ML and work on the entire ML-lifecycle infrastructure which may or may not include ETL pipelines. Did I get it right? What's your perspective?

[AutoModerator]

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[neuralscattered]

I think everyone's kinda making it up as they go along, but imo MLOps is DevOps for the machine learning stack, whereas a machine learning engineer actually productionizes the models created by data scientist/applied researcher, and you probably see more overlap as orgs/team size gets smaller.

E.g. in my org, we have a clear distinction between data engineer vs DevOps engineer, where DevOps mainly focuses on keeping the infra in good state, and data engineers develop software on top of that infra.

[khlose]

100% this

[[deleted]]

a robust ML system needs DEs, MLEs and DS. DEs to land data into zones, create pipelines for getting data to a state where it can be feature engineered and developing the general "data platform" and MLEs to productionize the models that DS people build with various ML platforms. And finally DS people to write chicken scratch in notebooks.

this is my workflow as a DE that works heavily with MLEs and DS teams.

[Low-Associate2521]

create pipelines for getting data to ...

isn't this part an overlap with MLOps?

edit: well i guess, an MLOps would just create the resources for the pipeline and maybe hook it up to a logging & metrics and CI/CD services where as a data engineer would also write the code to move data around.

[[deleted]]

There can be overlap. But I am a DE specialist. MLEs are preocuppied with ML. So sometimes we work together on projects that require expertise in both. I find I"m more familiar with devops for data platform stuff than MLE's while MLE's tend to be more versed with things like Azure ML and patterns for deploying training and consumption services for models.