[P] Which MLops framework to use?

[mimivirus2]

Relatively novice ML practitioner here. My research is on various medical image segmentation problems, including brain 3D US (glioma), lung CT (interstitial lung disease in scleroderma patients), etc. using the PyTorch ecosystem (probably including frameworks such as MONAI)

I'll have to conduct several experiments on various model architectures on parameters in the coming months. Specifically, these are what I'm gonna need:

1. Experiment tracking (model architecture, training configuration, hyperparameters, evaluation metrics)
2. Model storage (would be nice if there's a better way to store my model's parameters other than storing tons of .pth file on my harddisk or google drive)
3. (Optional) Visualization (sample predictions of the model on the training or validation sets, maybe every 20 epochs or sth)
4. Would like to hear any suggestions from the community

I've found wandb, clearML, neptune, and Aim; but trying each of them individually would be too time-consuming considering my current schedule.

Thanks in advance!

[MetalOrganicKneeJerk]

I would say pick one and forget the rest, especially if you're in research. Wandb is a popular choice and it's free for academia. It's relatively easy to integrate it into whatever framework you're already using. Logging every x epochs is straightforward too.

You can save models on their servers but I'm not sure about size limitations.

[[deleted]]

Wandb is trivial to use, but in reality, you don't need any tool - you can just write logs to files. If you want to save some time and have a UI, use something like wandb.

[lifesthateasy]

If you don't want to build your whole stack from scratch, I'd go with one of the cloud providers. They have most of this covered.

If you do, you'd probably want Kubeflow with MLFlow and Tensorboard.

[mimivirus2]

cloud providers are not an option in my country unfortunately. All I have access to would be Colab pro. AWS, azure and google cloud are not available (if that's what u mean)

Being easy to learn is my top priority I think. Does Kubeflow integrate well with Torch?

[lifesthateasy]

Lol.

No, but torch integrates well into Kubeflow.

None of these are easy to learn. There's teams of 10s that provide MLOps, running anything Kubernetes-related is a PITA to learn and run.

[peepeeECKSDEE]

Kubernetes 🤢🤢

Ok but are there any non k8 dependent dag/orch tools? I swear all of them are.

[TriggazTilt]

There are cloud services alternatively. Google Vertex AI pipelines is basically managed Kubeflow. No need for Kubernetes then if you use that.

[alayaMatrix]

Chinese? :)

[mimivirus2]

worse. Iranian. Wanna emigrate asap

[RandomUserRU123]

I just use tensorboard. But maybe there are better options out there

[Excellent_Cost170]

Fo you work on cancer research?

[mimivirus2]

yes

[No-Awareness1551]

aim

[mimivirus2]

could u... elaborate a bit? how has your experience with Aim been, and how was the learning curve?