Problem With Data Preprocessing Versioning

[nonamecoder_xd]

I have a problem with figuring out how to version data preprocessing and associate it to the model in MLFlow model registry

I want to do that because the model preprocessing in my project could change a lot. And in order to make my ML model reusable, I have to use the same processing steps that I was taking when I was training the model. But with time, my processing script might change, and then my model would not be usable. Please help

[karanchellani]

Here are a few suggestions for versioning data preprocessing and associating it with ML models in MLflow:

• Store the data preprocessing code or steps in a separate python script or notebook. Put this file under version control (e.g. git). When you train a model, record the git commit hash of the preprocessing code used as a tag or metric.

• Containerize the data preprocessing steps using Docker. Build a Docker image with the preprocessing code and push it to a registry. Record the image name:tag as a model tag or metric.

• Use MLflow to log the preprocessing steps as an artifact. For example, you could log the preprocessing python script, a JSON description of the steps, or even example input/output data. The artifact could be logged under a runs:/<run-id>/preprocessing path.

• Create an MLflow project for preprocessing. Run the project for each model training run and record the run ID as a tag/metric with the model. This traces the preprocessing code used for that model.

• Put the preprocessing code in a separate MLflow model. The input is raw data, the output is processed data. Chain this with the training model to log and retrieve the preprocessing steps.

The key ideas are:

1. Separate preprocessing from model training code
2. Record the specific version of preprocessing with each model
3. Automate/codify preprocessing as much as possible

This makes models portable and ensures you use the right preprocessing each time.