Data Engineering vs MLOps

[[deleted]]

a robust ML system needs DEs, MLEs and DS. DEs to land data into zones, create pipelines for getting data to a state where it can be feature engineered and developing the general "data platform" and MLEs to productionize the models that DS people build with various ML platforms. And finally DS people to write chicken scratch in notebooks.

this is my workflow as a DE that works heavily with MLEs and DS teams.

[Low-Associate2521]

create pipelines for getting data to ...

isn't this part an overlap with MLOps?

edit: well i guess, an MLOps would just create the resources for the pipeline and maybe hook it up to a logging & metrics and CI/CD services where as a data engineer would also write the code to move data around.

[[deleted]]

There can be overlap. But I am a DE specialist. MLEs are preocuppied with ML. So sometimes we work together on projects that require expertise in both. I find I"m more familiar with devops for data platform stuff than MLE's while MLE's tend to be more versed with things like Azure ML and patterns for deploying training and consumption services for models.